Hi. I'm applying for the Google Summer of Code 2013, this is my proposal for the registry merging project. I'll work on improving Wine's registry in its stability and simulation to solve frequently seen non user-friendly problems, which usually caused by that application or system settings are not stored and fetched correctly.
Project & My Understand: I'm most interested in the project "Registry - implement merging between HKEY_CLASSES_ROOT and HKEY_CURRENT_USER\Software\Classes". Its core task is to implement RegOpenUserClassesRoot, which returns a key token of merging view of HKCU & HKLM classes rather than present returning HKCR Classes. The difficulity is that as a view of two independent registry trees, how to treat conflicts and make changes to two seperated entrys when using HKCR as root, and how to achieve the function with no or least effect on other part of registry operation such as RegOpenKeyExX. I have some seed ideas that maybe not mature but I like solving problems initially from a burst port : Scheme 1. Registry operations based on hKey ( handle ) under HKEY_CLASSES_ROOT will be degeneration to full path in text, like "HKEY_CLASSES_ROOT\Subdir1", and try the operation after "HKEY_CLASSES_ROOT" replaced with "HKEY_CURRENT_USER\Software\Classes" and "HKEY_LOCAL_MACHINE\Software\Classes" in order, only the first success operation will be applied. This may satisfy the demand, both for query and modify. For details like the correct order, I'll make it out once get working on it. In this way, the HKCR key subtree does not exist in fact but only exists as a name. Scheme 2. After the base registry is initialized, build the HKCR's subkey trees with a new pulled in object type (say it 'symbol link') that directly point to the correct keys from HKLM or HKCU. Thus operation in HKCR tree are redirected to corresponding branch internally. When operating the HKCR itself, like creating new subkeys direct under HKCR, the operation is redirected to HKCU classes. This scheme seems more fluent but needs some tricks in the server part.
What I'll Do : First of all I'll do a lot of tests on pure winows registry to definite my questions and guesses, write one or more test reports on different version of windows, some of which is already being done now. Followed by I will design the algorithms and data structures and make a complete development planation. Then enjoying coding and debuging Wine. I love the procedure, during the longest part of this project, keep writing daily report. Got the plan finished, I will applying improvement. Finally strict tests will be done to show if I achieved the final goal and a conclude paper will be carefully submitted. I'd like not only settle down the main project, but also several ideas to make Wine better, which are found in my daily use of Wine, such as add support for the shell context menu in Wine explorer for many convenient shotcuts of some software like the famous 7zip. I'll do this once time allows.
Why Me : I myself is a faithful user of Wine. I know the thought of Wine and know not only basic usage but also a little implement of it. I understand Windows registry both Windows API pretty well from nearly 8 years coding. Have done much test and hooks in windows including RegXXX in advapi32.dll or lower level NtOpenKey in ntdll.dll. Further more, I used to do some kernel debug tests and hacks on registry, like hiding, protecting a subkey from regedit, resolving values from hive file rather than using API. These things are done under interests. I've been learning Windows API and OS theories from books and network like msdn for long. Ofcourse there will be no problem in registry knowledge. It's usual for me to write codes more than 1000 lines a day, for self use or for the competation ACM. I believe that I'm a careful man in codes and other things, I like discovering, guessing and proving. To prove I've developed a not too bad coding style, please refer to my submisions on codeforces http://codeforces.com/submissions/adnim (to view code click numbers under column "#"). And this may be a little evidence denoting that at least I've used git before, https://code.google.com/p/eryapass/ I know what should I do and what should not. I'm sure aware of the legal requirements. All of my tests will be done only in the black-box ways. No reverse work on Windows files is allowed and needed in Wine project, though I'm good at that. All codes will be written creatively and nothing will be copied from existing ones like WRK, RcactOS or something else. I love open source. I benefited much from the open source world and I'd be really happy to contribute to it one day. The last but most important, I'm much willing and I'm looking forward to throwing myself among the crowd of codes this entire summer, making maxium effort to improve Wine. I'd like schedule my personal activities to avoid conflicts with the development, meaning it won't be too busy in my schoolwork.
Wine & Me : In my childhood, I've been coding in F-Basic on Subor (a kind of TV-based computer made in China) when friends are playing football in yard, at about 8 years old. When I was 12 years old, i was building some little games to happy classmates, and have learned c and c++ 2 years latter, then x86 asm 2 more years latter. And have done some graduation-designs for college brothers before I got to university. Also I've developed some projects for local companies. I first installed Ubuntu in 2010 and gradually I'm interested in open source projects. I'm 21 years old now, studying software engineering in Shandong University of Science and Technology in China. I'm a really enthusiastic user of Wine, who has been using it from version 1.2 to 1.4, on ubuntu, debian and gentoo. Since the first day I apt-got it I could not help trying all applications I used in Windows to it and making them running stably, during which I have learned much about configuring Wine. Another day I began to emerging that and I was once deep interested in building my customed Wine and Wine based tests. I heard about GSoC last year but missed the time. This year, I decided that I will try my best to contribute, transform my ideas to powerful codes that people will run with satisfaction. Please do not hesitate to contact me.
Thanks. Guo Jian
Hi Guo Jian, just so you know, there's another application for the same project. This doesn't mean that yours can't be accepted, but it does impact your chances. Thank you for your interest. Good luck, --Juan
On Fri, May 3, 2013 at 1:05 AM, orzhvs orzhvs@gmail.com wrote:
Hi. I'm applying for the Google Summer of Code 2013, this is my proposal for the registry merging project. I'll work on improving Wine's registry in its stability and simulation to solve frequently seen non user-friendly problems, which usually caused by that application or system settings are not stored and fetched correctly.
Project & My Understand: I'm most interested in the project "Registry - implement merging between HKEY_CLASSES_ROOT and HKEY_CURRENT_USER\Software\Classes". Its core task is to implement RegOpenUserClassesRoot, which returns a key token of merging view of HKCU & HKLM classes rather than present returning HKCR Classes. The difficulity is that as a view of two independent registry trees, how to treat conflicts and make changes to two seperated entrys when using HKCR as root, and how to achieve the function with no or least effect on other part of registry operation such as RegOpenKeyExX. I have some seed ideas that maybe not mature but I like solving problems initially from a burst port : Scheme 1. Registry operations based on hKey ( handle ) under HKEY_CLASSES_ROOT will be degeneration to full path in text, like "HKEY_CLASSES_ROOT\Subdir1", and try the operation after "HKEY_CLASSES_ROOT" replaced with "HKEY_CURRENT_USER\Software\Classes" and "HKEY_LOCAL_MACHINE\Software\Classes" in order, only the first success operation will be applied. This may satisfy the demand, both for query and modify. For details like the correct order, I'll make it out once get working on it. In this way, the HKCR key subtree does not exist in fact but only exists as a name. Scheme 2. After the base registry is initialized, build the HKCR's subkey trees with a new pulled in object type (say it 'symbol link') that directly point to the correct keys from HKLM or HKCU. Thus operation in HKCR tree are redirected to corresponding branch internally. When operating the HKCR itself, like creating new subkeys direct under HKCR, the operation is redirected to HKCU classes. This scheme seems more fluent but needs some tricks in the server part.
What I'll Do : First of all I'll do a lot of tests on pure winows registry to definite my questions and guesses, write one or more test reports on different version of windows, some of which is already being done now. Followed by I will design the algorithms and data structures and make a complete development planation. Then enjoying coding and debuging Wine. I love the procedure, during the longest part of this project, keep writing daily report. Got the plan finished, I will applying improvement. Finally strict tests will be done to show if I achieved the final goal and a conclude paper will be carefully submitted. I'd like not only settle down the main project, but also several ideas to make Wine better, which are found in my daily use of Wine, such as add support for the shell context menu in Wine explorer for many convenient shotcuts of some software like the famous 7zip. I'll do this once time allows.
Why Me : I myself is a faithful user of Wine. I know the thought of Wine and know not only basic usage but also a little implement of it. I understand Windows registry both Windows API pretty well from nearly 8 years coding. Have done much test and hooks in windows including RegXXX in advapi32.dll or lower level NtOpenKey in ntdll.dll. Further more, I used to do some kernel debug tests and hacks on registry, like hiding, protecting a subkey from regedit, resolving values from hive file rather than using API. These things are done under interests. I've been learning Windows API and OS theories from books and network like msdn for long. Ofcourse there will be no problem in registry knowledge. It's usual for me to write codes more than 1000 lines a day, for self use or for the competation ACM. I believe that I'm a careful man in codes and other things, I like discovering, guessing and proving. To prove I've developed a not too bad coding style, please refer to my submisions on codeforces http://codeforces.com/submissions/adnim (to view code click numbers under column "#"). And this may be a little evidence denoting that at least I've used git before, https://code.google.com/p/eryapass/ I know what should I do and what should not. I'm sure aware of the legal requirements. All of my tests will be done only in the black-box ways. No reverse work on Windows files is allowed and needed in Wine project, though I'm good at that. All codes will be written creatively and nothing will be copied from existing ones like WRK, RcactOS or something else. I love open source. I benefited much from the open source world and I'd be really happy to contribute to it one day. The last but most important, I'm much willing and I'm looking forward to throwing myself among the crowd of codes this entire summer, making maxium effort to improve Wine. I'd like schedule my personal activities to avoid conflicts with the development, meaning it won't be too busy in my schoolwork.
Wine & Me : In my childhood, I've been coding in F-Basic on Subor (a kind of TV-based computer made in China) when friends are playing football in yard, at about 8 years old. When I was 12 years old, i was building some little games to happy classmates, and have learned c and c++ 2 years latter, then x86 asm 2 more years latter. And have done some graduation-designs for college brothers before I got to university. Also I've developed some projects for local companies. I first installed Ubuntu in 2010 and gradually I'm interested in open source projects. I'm 21 years old now, studying software engineering in Shandong University of Science and Technology in China. I'm a really enthusiastic user of Wine, who has been using it from version 1.2 to 1.4, on ubuntu, debian and gentoo. Since the first day I apt-got it I could not help trying all applications I used in Windows to it and making them running stably, during which I have learned much about configuring Wine. Another day I began to emerging that and I was once deep interested in building my customed Wine and Wine based tests. I heard about GSoC last year but missed the time. This year, I decided that I will try my best to contribute, transform my ideas to powerful codes that people will run with satisfaction. Please do not hesitate to contact me.
Thanks. Guo Jian
Hi Juan, Thank you for your reply. Sincerely sorry for my late application, you know that's really not pleasing. But I think that increase my enthusiasm at the same time. I will keep focus and spend more time on it as I'm just getting start in the community. I love it even more when it seems to be a challenge :)
regards. Guo Jian
2013/5/4 Juan Lang juan.lang@gmail.com
Hi Guo Jian, just so you know, there's another application for the same project. This doesn't mean that yours can't be accepted, but it does impact your chances. Thank you for your interest. Good luck, --Juan
On Fri, May 3, 2013 at 1:05 AM, orzhvs orzhvs@gmail.com wrote:
Hi. I'm applying for the Google Summer of Code 2013, this is my proposal for the registry merging project. I'll work on improving Wine's registry in its stability and simulation to solve frequently seen non user-friendly problems, which usually caused by that application or system settings are not stored and fetched correctly.
Project & My Understand: I'm most interested in the project "Registry - implement merging between HKEY_CLASSES_ROOT and HKEY_CURRENT_USER\Software\Classes". Its core task is to implement RegOpenUserClassesRoot, which returns a key token of merging view of HKCU & HKLM classes rather than present returning HKCR Classes. The difficulity is that as a view of two independent registry trees, how to treat conflicts and make changes to two seperated entrys when using HKCR as root, and how to achieve the function with no or least effect on other part of registry operation such as RegOpenKeyExX. I have some seed ideas that maybe not mature but I like solving problems initially from a burst port : Scheme 1. Registry operations based on hKey ( handle ) under HKEY_CLASSES_ROOT will be degeneration to full path in text, like "HKEY_CLASSES_ROOT\Subdir1", and try the operation after "HKEY_CLASSES_ROOT" replaced with "HKEY_CURRENT_USER\Software\Classes" and "HKEY_LOCAL_MACHINE\Software\Classes" in order, only the first success operation will be applied. This may satisfy the demand, both for query and modify. For details like the correct order, I'll make it out once get working on it. In this way, the HKCR key subtree does not exist in fact but only exists as a name. Scheme 2. After the base registry is initialized, build the HKCR's subkey trees with a new pulled in object type (say it 'symbol link') that directly point to the correct keys from HKLM or HKCU. Thus operation in HKCR tree are redirected to corresponding branch internally. When operating the HKCR itself, like creating new subkeys direct under HKCR, the operation is redirected to HKCU classes. This scheme seems more fluent but needs some tricks in the server part.
What I'll Do : First of all I'll do a lot of tests on pure winows registry to definite my questions and guesses, write one or more test reports on different version of windows, some of which is already being done now. Followed by I will design the algorithms and data structures and make a complete development planation. Then enjoying coding and debuging Wine. I love the procedure, during the longest part of this project, keep writing daily report. Got the plan finished, I will applying improvement. Finally strict tests will be done to show if I achieved the final goal and a conclude paper will be carefully submitted. I'd like not only settle down the main project, but also several ideas to make Wine better, which are found in my daily use of Wine, such as add support for the shell context menu in Wine explorer for many convenient shotcuts of some software like the famous 7zip. I'll do this once time allows.
Why Me : I myself is a faithful user of Wine. I know the thought of Wine and know not only basic usage but also a little implement of it. I understand Windows registry both Windows API pretty well from nearly 8 years coding. Have done much test and hooks in windows including RegXXX in advapi32.dll or lower level NtOpenKey in ntdll.dll. Further more, I used to do some kernel debug tests and hacks on registry, like hiding, protecting a subkey from regedit, resolving values from hive file rather than using API. These things are done under interests. I've been learning Windows API and OS theories from books and network like msdn for long. Ofcourse there will be no problem in registry knowledge. It's usual for me to write codes more than 1000 lines a day, for self use or for the competation ACM. I believe that I'm a careful man in codes and other things, I like discovering, guessing and proving. To prove I've developed a not too bad coding style, please refer to my submisions on codeforces http://codeforces.com/submissions/adnim (to view code click numbers under column "#"). And this may be a little evidence denoting that at least I've used git before, https://code.google.com/p/eryapass/ I know what should I do and what should not. I'm sure aware of the legal requirements. All of my tests will be done only in the black-box ways. No reverse work on Windows files is allowed and needed in Wine project, though I'm good at that. All codes will be written creatively and nothing will be copied from existing ones like WRK, RcactOS or something else. I love open source. I benefited much from the open source world and I'd be really happy to contribute to it one day. The last but most important, I'm much willing and I'm looking forward to throwing myself among the crowd of codes this entire summer, making maxium effort to improve Wine. I'd like schedule my personal activities to avoid conflicts with the development, meaning it won't be too busy in my schoolwork.
Wine & Me : In my childhood, I've been coding in F-Basic on Subor (a kind of TV-based computer made in China) when friends are playing football in yard, at about 8 years old. When I was 12 years old, i was building some little games to happy classmates, and have learned c and c++ 2 years latter, then x86 asm 2 more years latter. And have done some graduation-designs for college brothers before I got to university. Also I've developed some projects for local companies. I first installed Ubuntu in 2010 and gradually I'm interested in open source projects. I'm 21 years old now, studying software engineering in Shandong University of Science and Technology in China. I'm a really enthusiastic user of Wine, who has been using it from version 1.2 to 1.4, on ubuntu, debian and gentoo. Since the first day I apt-got it I could not help trying all applications I used in Windows to it and making them running stably, during which I have learned much about configuring Wine. Another day I began to emerging that and I was once deep interested in building my customed Wine and Wine based tests. I heard about GSoC last year but missed the time. This year, I decided that I will try my best to contribute, transform my ideas to powerful codes that people will run with satisfaction. Please do not hesitate to contact me.
Thanks. Guo Jian
Have been working hard to get familiar with the workflow in wine, by reporting bugs (bug 33542 33550 and 33558) and sending patches (patch 96154 and 96157) to wine. I got more accustomed to the community to find I like here and I will keep working in wine even after GSoC.
Working progress :
I've concluded that all the registry operations : *create_key *delete_key *delete_value =enum_key =enum_value -flush_key -get_key_security -load_key -notify_change_key_value =open_key -override_predef_key =query_info_key =query_multipe_values =query_value -replace_key -restore_key -save_key -set_key_security *set_value -unload_key [ - operations that are not very important and less used, put them aside at first ] [ * operations that change the contents ] [ = operations that only read contents ] In my previous tests I found that create_key & delete_key of HKCR works like simply redirecting to HKCU or HKLM (depending on how their subkeys conflict). So I guess that all the operations have such feature. The next thing I need to write tests for : - Where does each write-operations redirect to when there's [no subkey|both subkeys] in HKCU & HKLM. - Where does each read-operations redirect to when HKCU & HKLM conflicts. There is one thing that I can make sur, when it does not conflict in HKCU & HKLM, operations are redirected to the only one.
Another guess : open_key (including opening from create_key) will be a special operation, it should not return a handle directly based on HKCU or HKLM, or further operation on the handle will be treated as opened directly from the source. So it may be necessary to define a new type of key handle.
Updated time table:
Start Date End Date Event 3rd, May 12th, May Work on some basic knowledge (bugs, patches, winedbg debugchannel), learning workflow of wine. 13th, May * 16th, Jun Write full tests and test reports. Make puzzles clear. Design the main algorithm. 28th, May 24th, Jul Implement almost all of the registry merging feature. Submit as patch and track feedbacks. 25th, Jul 31st, Jul Finish and submit the Mid-term evaluations. 1st, Aug 31st, Aug Implement the rest part of our scheme. Apply changes by patches. 1st, Sep 22th, Sep Cleanups. Write the final evaluations. 23th, Sep / Submit to Google after some modification from the mentor.
Note that the 2nd and 3rd part have time intersection beacause I think each of the two is based on another.
I decided to rewrite the tests in orderliness. Today I have done the tests on write-operations ( http://newtestbot.winehq.org/JobDetails.pl?Key=931 ) . And ome of my assumptions has been confirmed:
All the operations that change hkcr should prefer hkcu to hklm. That means if the _full_path_of_the_operation_ can be found in hkcu it act as mapping to hkcu, otherwise to hklm. _full_path_of_the_operation_ : for create_key and delete_key, it's the path of the handle plus the subkey path. for set_value and delete_value, it's the path of the handle. For example, set new value on 'val1 of hkcr/test1 affects hkcu when there are hkcu/software/classes/test1 and hklm/software/classes/test1, but affects hklm when there's only hklm/software/classes/test1.
Another thing that I confirmed is that handle opened from hkcr is not the same as that from hkcu or hklm, even when there's only one such key in hkcu or hklm. Because further operations on the handle change the other when the 'path match' situation is meet. For example, make sure there is hkcu/software/classes/key1 but not hklm/software/classes/key1, after creating a new key subkey2 with the handle opened from hkcr/key1, it will be found as hklm/software/classes/key1/subkey2 rather than hkcu. Obviously operationg on a handle opened from hkcu/software/classes/key1 does not affect hklm, so there must be something necessary to do to tell the mapped key handle from oridinal one. My preliminary design: open_key for hkcr subkeys should return a new type handle - key_view, which have one field : key path relating to classes, all operations on the handle will be dynamic decided which branch to redirect to. As far as I know from my previous developing windows drivers, kernel and services run codes as system user, so the handle may be confused when using by different user, so I will test what happen when using a dumped key handle of hkcr subkeys across different users. To decide whether a more field 'owner uid' is needed.
I'm going to test on read-operations soon, after this I'll start writing the test reports.
I just found that the REG_OPTION_VOLATILE of create_key in hkcr may have some tricks. Not surprisingly found a strange situation when testing on windows. See my test here please : http://newtestbot.winehq.org/JobDetails.pl?Key=932 This happens as following : First create hkcu/software/classes/key1, then create the hkcr/key1/subkey with REG_OPTION_VOLATILE. RegEnumKey on hkcr will give two 'key1'. I tried to use RegQueryInfoKeyA to count them, also there are more than there should be. Why there are two duplicated keys? May it be a bug of windows? If it is, should we design our algorithm to avoid or treat it as a normal feature?
And a new test is here http://newtestbot.winehq.org/JobDetails.pl?Key=934
2013/5/15 Guo Jian orzhvs@gmail.com:
I just found that the REG_OPTION_VOLATILE of create_key in hkcr may have some tricks. Not surprisingly found a strange situation when testing on windows. See my test here please : http://newtestbot.winehq.org/JobDetails.pl?Key=932 This happens as following : First create hkcu/software/classes/key1, then create the hkcr/key1/subkey with REG_OPTION_VOLATILE. RegEnumKey on hkcr will give two 'key1'. I tried to use RegQueryInfoKeyA to count them, also there are more than there should be. Why there are two duplicated keys? May it be a bug of windows? If it is, should we design our algorithm to avoid or treat it as a normal feature?
Hi Guo,
On Tue, May 14, 2013 at 1:32 PM, Guo Jian orzhvs@gmail.com wrote:
I just found that the REG_OPTION_VOLATILE of create_key in hkcr may have some tricks. Not surprisingly found a strange situation when testing on windows. See my test here please : http://newtestbot.winehq.org/JobDetails.pl?Key=932 This happens as following : First create hkcu/software/classes/key1, then create the hkcr/key1/subkey with REG_OPTION_VOLATILE. RegEnumKey on hkcr will give two 'key1'.
I tried to use RegQueryInfoKeyA to count them, also there are more
than there should be. Why there are two duplicated keys? May it be a bug of windows? If it is, should we design our algorithm to avoid or treat it as a normal feature?
That's interesting. Windows might allow both a volatile and non-volatile key of the same name. Since the volatile one is kept in-memory, any values that exist in it will be discarded eventually, so any inconsistency that arises isn't the kernel's problem. That'd be my guess, at any rate: even though such a behavior has the potential to confuse application developers, since it's not the kernel's problem, it's probably an overlooked quirk.
As far as what to do with it: this is one of those nuanced, i.e. hard to decide, things with our tests.
In general, we assume that Windows's behavior is the correct behavior, unless we have strong reason to assume otherwise. We've encountered many times that even what appear to be bugs end up being relied upon by application developers.
In this case, though, I think you might have stumbled across something that's relatively rare, and unless there's an application that depends on this behavior, it might be worth ignoring for now.
Hope that helps. --Juan
Sorry to bother. Forgot CC to wine-devel.
In this case, though, I think you might have stumbled across something that's relatively rare, and unless there's an application that depends on this behavior, it might be worth ignoring for now.
It's in rear case that applications use volatile key under hkcr. Won't waste time on the not so important detail if no further effects. Thank you.
Tests for read-operations has been done. http://newtestbot.winehq.org/JobDetails.pl?Key=942 All the results are expected: 1. query_value prefer values in hkcu. 2. query_key_info gives the count of subkeys and values distincted. 3. enum_value & enum_key combine result from hkcu & hklm and return in sorted order I've looked into server/registry.c to find the subkeys are stored in alphabet order and is located by binary search. So when enumerating subkeys or values from hkcr, the two branches hkcu and hklm can be combined using a algorithm like part of merge_sort.
Had hard time testing what happens using a dumped key handle of hkcr from another user, as mentioned in last mail. Steps in my test: 1. make sure hkcu/software/classes/subkey1 exist in hkcu of both user1 and user2. 2. program1 of user1 set 'val1' in hkcu/software/classes/subkey1 to 'user1' 3. program1 opens a handle to hkcr/subkey1, print it out then wait but not exit. 3. program2 of user2 dumps the handle from program1 to program2, set value 'val1' to 'user2' with that handle. 4. program1 continues, open hkcu/classes/subkey1. query and print out current value of 'val1'. Here we care what value is eventually set to 'val1' in user1. As we know a handle is associated with a kernel object. dumped handle points to same object as original. - If the final value is 'user1', that denotes operations by user2 with hckr key objects created from user1 have no effect on user1. - If the value is 'user2', operations from another user changes content from owner user, obviously the key-view is associated with the own user I'm considering to add a field 'owner' in my key view object. The test code is attached, and output is as following, note that the second program start running when the first paused.
runas /user:user1 "testcls 1"
program1 on user1 has pid : 2092 'val1' set to 'user1' on user1 handle of subkey1 opened from hkcr : 000007D2 come back after runing program2 as user2 press any key to continue . . 'val1' on user1 is 'user2' now
runas /user:user2 "testcls 2"
input pid of program1 2092 input handle of subkey1 in program1 7d2 dumped handle on user2 : 000007B0 'val1' set to 'user2'
The final result turned out to be 'user2'. As expected, isn't it?
Till now, most of my test work has finished. I'm going to make out a stage conclusion soon then I'll come to the main point - the implement of merging.
-
Guo Jian -
Juan Lang -
orzhvs