| Author | Message | |||||||||||||||
     Ross Johnson Username: ross_winpro_net Registered: N/A |
V. 13.7 SR-6 I believe the following to be an error between Windows and WinHex (but may be fixable by WinHex?) If a folder name ends with one or more spaces (e.g. "mystuff "), WinHex will not recover any content for that folder (i.e. files and subfolders). Even worse, any remaining recovery aborts at this point without any error message, but does display the normal stats as if all went well. Depending on where in the recovery any such folder may exist, you may not notice the aborted recovery unless you compare source and destination stats (which I recommend as best practice). ** additional notes: The Directory Browser does display the subfolders and files. Individual items selected inside such a named folder will Recover/Copy if "Recreate full orig. path" is not checked. The destination is an NTFS hard drive. WinHex is probably passing the name, as found, to Windows (because if WinHex were truncating the name then it would also do so for the remaining content, thus avoiding the situation); Windows then truncates the name when creating the destination folder (i.e. removes any trailing spaces), then WinHex tries to send the source folder's remaining content to the destination folder (now named without the spaces) but it does not exist as named, so it does not Recover/Copy that content or any other remaining content from the source. With limited testing, I have been able to repeat the results. Can anyone else can confirm? I suspect this may occur under other conditions still to be tested (e.g. other trailing characters that Windows considers invalid for folder names?) Thank you, Ross@WinPro.net | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
A little more research suggests that a source filename ending with a period may also cause this issue. Needs confiriming. Thank you, Ross@WinPro.net | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
My apologies, here is a correction and some additions to the original post: The "Messages" box does display a relevant message such as: Cannot create "C:\`test.rcv\User\Personal\ My information \Today". Make sure the folder exists and the file is not write-protected. The system cannot find the path specified. In this example, there is a space after "My information". However, if the folder name with the trailing space is the last folder in the path, the space will not be obvious in such a message. Also, the message does not indicate that Recover/Copy aborted prematurely. another note: a backslash within a source filename will also cause a similar destination issue. In this case WinHex expects there to be a destination folder named with the part of the filename that precedes the back slash. WinHex seems to handle a colon ":" in a filename AOK by replacing it with an underscore when sending it to the destination (is this correct?) If so, then this could be the solve for any other invalid character? Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
I tried (under Windows XP), and found that if Windows truncates a directory's name and WinHex consequently cannot copy the contents of that directory (as the expected path does not exist), WinHex correctly issues warnings and displays the results correctly. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
Wish (major) item: When "Recursive selection statistics" is enabled (selection statistcs are displayed in blue on the right side of the View bar). A 100% successful Recover/Copy of this selection will display a message box with the same statistics. I know that these can be visually compared (best practice). However, since the nature of the software and the business is "recovery", I wish for WinHex to be able to automatically warn when these do not match. Most importantly, perhaps, even when "Recursive selection statistics" is not enabled; which is not easily visually compared. Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
For warnings about the recover/copy process please see the messages window. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
> I tried (under Windows XP) OK, I was originally using a W2k host. ... I just tested an XP host. It still creates the same issue. ---- Perhaps you need more details?: I originally noticed this situation on a source image file. However, I also tested by using a second source; an NTFS physical HD. Synopsis of test on Physical HD: Test folders had 3 or more folders all with recoverable content (I created folders and subfolders each with a different size to ease the spotting of discrepancies). Unchecked Security option "open files thru operating system", I changed the last character in an existing folder (with sub content) to a space (0x20, using WinHex), rebooted, added this HD to a case, selected all content of this HD, used Recover/Copy on the selection, checked "Recreate full orig. path". The resulting "Messages" window correctly noted the failure to recover anything in the renamed folder (and the empty destination folder was created without the space). At this point it will appear that recovery completes AOK (excepting the content listed in the 'Messages" window), but further vigilant examination proves different. Because of the un-recovered folder (listed in the "Messages" window), the final recovery stats will not match the recursive stats (of course, as expected, but misleading) ... further examination shows that the quantity of the content in the failed folder was not accounting for the full difference between the recursive stats and the the final stats. It appears that WinHex is quietly aborting further recovery of the remaining selections (i.e. no message). But you would have to add the Folders/Files/MB quantities of the resulting stats to the quantities of the items listed in the "Messages" window (calculating that manually can be a chore, which is probably why this is being missed), then compare those combined results with the recursive selection statistics (if enabled) [isn't this fun?]. There will be a mismatch (unless the folder with the changed name is the last item in the Directory Browser). Try repeating and sorting the DB to put the suspect folder in the middle of the recovery or at the beginning. The results will change even they should not (if just the "Messages" window items are being excluded from the recovery and nothing else, then sort order should not matter). In my testing today I used 2 host systems, W2k on one system, XP on two systems, 2 sources. The "Messages" window does contain a list of content that was definitely not recovered. But under the conditions laid out above, the "Messages" window can also fail to list additional objects that were not recovered (but are very easily recoverable) (and this undocumented amount of missed objects can range from very little to all else selected depending on the DB sort order). Summary: from the same source with the same selection of objects, recovery results can vary widely simply by having a source folder (or more) with a name not acceptable to the destination and then changing the sort order, yet the "Messages" Window lists the same missed objects each time. Additionally, I retested the source image by trying various folder selections and noticed more mismatches between the recursive stats and the the final stats (i.e. mismatches that exceeded the inclusion of the Messages window content). I traced down the back slash character as being another cause for WinHex to quietly abort prematurely (and here I noticed that the colon was being converted OK by WinHex and therefore was hoping that the space, period, backslash and any other could be easily added to WinHex?) Please explain if I am misunderstanding some naming rules or procedures in this issue, but I truly believe that there is a problem here that (without vigilance and manual effort) allows for failure of maximum recovery. BTW: "Recreate full orig. path" on/off can make a difference depending on the path position of the suspect folder (e.g. is the suspect folder a parent/child/same-level of your recovery selection). That, combined with the DB sort order allows for many many combos to test, yet the results may not be blatantly obvious without some work (of which I have done some). If the renamed folder is empty, then the issue is avoided because WinHex is not trying to fill it on the destination. So make sure your test folder has content. WinHex has no problem navigating these folders or displaying content via Gallery or Preview (only trouble is Copy/Recover). > selection statistics are displayed in blue on the right side of the View bar is just for reference (may not be blue if filtering is off) It has been a few hours since I started this post. I have done more testing and I can confirm this to be consistent across machines and OS's. I think it is serious and worth resolving, can anyone else confirm? My next test is to review recent cases to see if any clients failed to receive all available recovery. I recall once in the last three months where a DVD had far less than the DB said should be there, so I had to manually select subfolders for recovery on the second try. But it will be the smaller losses that will hurt the most. Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
If Windows does not accept the creation of a file in the output directory, X-Ways Forensics stops the recover/copy process and issues a warning. In recent v13.8 XWF Beta versions, X-Ways Forensics tries to continue with the next file. | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
...and so does v13.7 SR-7 now. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
It is getting better ... > ...and so does v13.7 SR-7 now Yes, SR-7 does now continue with the next file (and/or folder), thank you. SR-7 resolves the Directory Browser sort order issue for subfolders, thank you. SR-7 also now recovers files from the same level as the objectionable foldername. Without any warning or message, all such selected files were being skipped (for any sort order). I am assuming then that files are always recovered after any folders (at each level)? However, SR-7 still does not recover anything from within the source folder that has the NTFS objectionable name (i.e. under specific conditions, when "Recreate full orig. path" is on, WinHex is still sending the objectionable folder name to Windows). Yet, when "Recreate full orig. path" is off WinHex is able to send the corrected folder name to windows and successfully recover all objects from the subfolders. -------------------------------- FYI re: > X-Ways Forensics stops the recover/copy process and issues a warning There was never any warning (message) that X-Ways Forensics had stopped the recover/copy process. The only warnings during the Recover/Copy effort were like this one: Cannot create "C:\`test.rcv\User\Personal\ My information \Today". Make sure the folder exists and the file is not write-protected. The system cannot find the path specified. (Nor any warning that Recover/Copy skipped the remaining selected objects.) As announced, v13.7 SR-7 has "* Minor fixes and improvements, including the Recover/Copy command". I concur and again I thank you for the effort (and I look forward to any major improvements). Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
> I am assuming then that files are always recovered after > any folders (at each level)? Should depend on whether directory/file grouping is active. > However, SR-7 still does not recover anything from within > the source folder that has the NTFS objectionable name If the path where such files are to be saved does not exist, then they are not saved in that path, and a warning is issued. > There was never any warning (message) that X-Ways > Forensics had stopped the recover/copy process. I think there are many error messages or warnings that do not tell whether the process that triggered them was aborted or continued, so that's not so unusual, I think. IMO, after such a serious path problem is reported, it is the task of the user to verify whether the result meets his/her needs. How exemplary that X-Ways Forensics displays the exact recover/copy final results in such a case, and these results and the case log leave no doubt about how much and what was actually recovered/copied. FYI: If not the initial creation of an output file fails as in the scenario you describe, but saving it completely fails, e.g. because free drive space or quota is up, then X-Ways Forensics v13.7 SR-7 will still totally stop the recovery (and rightfully so, IMO), outputs the error message, and tells you how much it has recovered. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
> If the path where such files are to be saved does not exist, then they are not saved in that path, and a warning is issued. Yes, but in this case, it is the fault of WinHex that such a destination path does not exist (not the fault of the destination file system). Because when WinHex tries to fill the destination folder, it is no longer asking for the same path that it caused to be created! The problem is that with WinHex Recover/copy (using the same selection of objects), I can choose one set of options and the destination path is correctly created and filled; or I can choose another set of options and WinHex goes as far as creating the destination folder (which has the exact same name as created in effort one) but then fails to fill it because suddenly the destination folder no longer exists! Yet it was created through WinHex Recover/Copy, and any subsequent effort to fill it via Recover/copy is not handling the source name as it did when creating the folder or as it did with the first set of options. Since all of that was rather verbose, here is a pair of short example descriptions: Using two different sets of Recover/copy options and a source folder named "test " (Note the space after 'test') ... Example 1. Recover/copy creates the destination folder called "test" (without the trailing space), then fills it with the content from "test " 100% AOK Example 2. Recover/copy creates a destination folder called "test" (without the trailing space), then fails to fill it with the content from "test " but generates a message like this for each contained object: Cannot create "C:\test \myfile.txt". Make sure the folder exists and the file is not write-protected. The system cannot find the path specified. Note the space after 'test' in the message. This is the problem! Plus needing to remember all the combinations of options that will cause it or not cause it. In both examples WinHex knows the source is named "test " but causes Windows (NTFS) to create a destination folder called "test". To me this makes sense and is just fine. However, in example 1, WinHex is sending recovery to a folder called "test" but in example 2 WinHex is sending recovery to folder called "test " which cannot exist on an NTFS system nor was it caused to be created by WinHex. To me this does NOT make sense and is not good I am not asking for it to be explained or defended; rather I would like to know if it will be left as is or altered so that I can adjust my Recover/copy procedures accordingly? Fortunately I have found a limited work around that will help me avoid pouring through the many error messages or warnings that do not tell whether the process that triggered them was aborted or continued ... which is not desirable. >IMO, after such a serious path problem is reported, it is the task of the user to verify whether the result meets his/her needs. I agree 100% and yes, the results did not meet my needs, thus the posts. > How exemplary that X-Ways Forensics displays the exact recover/copy final results in such a case, and these results and the case log leave no doubt about how much and what was actually recovered/copied. Again, I agree 100%. The final results left no doubt as to how much was actually recovered (simply checking the properties of the destination folder would tell me the same, i.e. files/folders/megabytes). However, it is the data NOT recovered that is laborious to assess. Since the "Messages" window was not including selected objects that were being skipped, I had to make an exemplary effort to calculate and manually recover all skipped objects plus manually recover the many objects listed in the "Messages" window (prior to the work around). FYI: no issues with destination drive space. Tested with Multiple destination HD's with plenty of room and confirmed healthy (best practice always). Some tests involved less than 100MB of data. Class dismissed. Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
> Note the space after 'test' in the message. This is the problem! I know. > I am not asking for it to be explained Nonetheless: > In both examples WinHex knows the source is named "test " > but causes Windows (NTFS) to create a destination folder > called "test". That Windows creates a directory "test" although WinHex asked it to create a directory "test " is unknown to WinHex. Windows itself decides to make that change and does not somehow inform WinHex of it. > rather I would like to know if it will be left as is or > altered so that I can adjust my Recover/copy procedures > accordingly If you know a definite, comprehensive list of rules that determine which and how directory names are arbitrarily changed by Windows (trailing spaces, trailing dots, maybe more), I would consider implementing in WinHex a way to work around this. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
> That Windows creates a directory "test" although WinHex asked it to create a directory "test " is unknown to WinHex. 1a. If "test" is unknown to WinHex, then how does WinHex manage to always correctly fill the destination "test" folder in the first example? 1b. Via the same "arbitrary" Windows name change? 2. Should WinHex really be asking Windows to create a folder called "test "? Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
Sorry, I don't have the time to further explain and defend against your wrong accusations ("it is the fault of WinHex that..."). Different methods/ways to recover files/directories may have different effects. 2. Yes, WinHex should really be asking Windows to recreate directories with their original names, or else you would probably be (rightfully) among the first to complain that it deliberately changes the names. My idea was if comprehensive rules could be found to work around the name changes that these could be implemented and documented as a single exception where the original names are altered. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
As a long time user of WinHex, I (and others), have requested and seen implemented, the ability to recover from additional File Systems (e.g. HFS, CDFS, etc.). And I thank X-Ways, as it would be difficult to perform forensic or standard recovery without such. WinHex can now read and recover from many more file systems than it can write to (FATxx & NTFS only?) Because WinHex can, and in many cases must, perform cross platform (i.e. cross file system) recovery; I would expect it to be fully able to translate/convert between the differing naming conventions (of the file systems supported by WinHex)? This is not happening now. (And because the "Messages" window was not necessarily listing all skipped objects, many WinHex users may be unaware of those skipped recoveries.) We have the same naming issue when burning DVDs of the recovery, but all burning software we use will notify us when there is a naming/path issue and allow for acceptable options such as character substitution or to pack recovery in an archive that preserves names/paths. > My idea was if comprehensive rules could be found ... There are many published tables that display and compare the differing naming conventions used by the file systems that WinHex can now recover. If you would like, I could dig one up for you? But it really seems like WinHex could simply focus on intercepting all characters not allowed by the naming convention of the destination, regardless of the source? However, does WinHex even know the destination file system in advance [i.e. FAT or NTFS] so that it can adapt for their naming differences? (For best practice, we always use NTFS anyway for other reasons also). > 2. Yes, WinHex should really be asking Windows to recreate directories with their original names, or else you would probably be (rightfully) among the first to complain that it deliberately changes the names. Do you mean like this: Using WinHex to Recover/copy "test > 2" results in a deliberately changed name on the destination of "test _ 2", it has been doing this for awhile, as it should. Why would anyone complain about this? No one has, I have not. It makes sense for this to happen when the destination cannot handle the source naming convention and I very much doubt WinHex will directly output to file systems other than FAT or NTFS anytime soon? Perhaps across a network though? My "complaint" is that WinHex is not doing the same for a few other source naming convention variations that are also not supported by NTFS or FAT, such as the space at the end of a name, that is all. I do not think I can be any more clear, can anyone help explain/translate? --------- Regarding these cases where the source file system contains naming conventions not allowed in the destination (NTFS); I have never seen any directory names arbitrarily changed. The changes I have seen have a pattern based on settings selected in Recover/copy. After an excessive investigative effort, I can now recreate (in WinHex) many exact Recover/copy option setting combinations that will always result in a quiet failure; or other combinations (selective) that will always succeed. The only thing arbitrary is the choice of options in Copy/recover made by the user. If WinHex is consistently passing the exact same path parameters in each example to Windows (i.e. "test ") and Windows is arbitrarily changing the folder name, then how does Windows know of the exact options selected in Recover/copy that cause success of failure? I would surmise that WinHex is sending different information in each case. Hmm, arbitrarily consistent? Now there is a conundrum. I find it hard to believe that in Example 1 above, Windows is successfully handling a source name of "C:\test \myfile.txt", when it is failing in example 2? There must be some difference in what WinHex is sending Windows in example 1 to allow for success (e.g. "C:\test\myfile.txt" or "C:\..\myfile.txt" or ??)? This is posted as an FYI for all users, Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
FYI: Directories/files with characters considered illegal by Windows (these characters are well documented in Windows) cannot be created (the Windows functions refuse to do that and return an error), so WinHex has to substitute these characters, whereas directories/files names with all legal characters can be created but are sometimes altered by Windows without returning an error, so WinHex rightfully proceeds as if all is well. | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
BINGO > FYI: Directories/files with characters considered illegal by Windows (these characters are well documented in Windows) cannot be created (the Windows functions refuse to do that and return an error), so WinHex has to substitute these characters, whereas directories/files names with all legal characters can be created but are sometimes altered by Windows without returning an error, so WinHex rightfully proceeds as if all is well. I think you understand now (?) Which is why I suggested (Programming option-1): > WinHex could simply focus on intercepting all characters not allowed by the naming convention of the destination A space character is not allowed in the final position in the Windows naming convention (even if no error is returned). here are 3 more (just suggestions) programming fixes ... Programming Option-2 (problematic though): WinHex could use a small LIFO buffer to retain the most recent path information sent to Windows then if WinHex receives a message such as: Cannot create "C:\test \myfile.txt" It is likely that the last buffer entry is the culprit that WinHex can review and use to compare and adjust. Programming Option-3: A menu item routine to predetermine if the source has Cross platform naming convention conflicts with the destination platform. Then allow for the results to be used in the actual recover. Programming Option-4 (my favorite): When WinHex receives a message such as Cannot create "C:\test \myfile.txt" WinHex can then check the (just used) path for naming convention violations (i.e. not just illegal characters). ----- At MSDN (a Microsoft developers network, for those who write programs for the Windows platform) http://msdn2.microsoft.com/en-us/library/aa365247.aspx, you can find a list of 8 rules that ... "enable applications to create and process valid names for files and directories regardless of the file system:" Rule # 8 is: "Do not end a file or directory name with a trailing space or a period. Although the underlying file system may support such names, the operating system does not." The other 7 Microsoft Programing rules should satisfy your request for: > If you know a definite, comprehensive list of rules that determine which and how directory names are arbitrarily changed by Windows (trailing spaces, trailing dots, maybe more), I would consider implementing in WinHex a way to work around this. (However, I interpreted 'rules that determine ... arbitrarily' as contradictory, I thought you were being facetious, if not sarcastic?) I hope this meets your request? The list is definitive (from Microsoft, for programmers) and comprehensive (but not arbitrary) and it does include trailing spaces, trailing dots, and more. It also includes other naming restrictions for Windows that may be allowed on other file systems (e.g. some reserved names). This is as far as I can go on this subject, if WinHex is not to utilize these rules then at least WinHex users can use the link to review why some of the objects are not being recovered during cross platform recovery. ------- I have also started a WinHex script (based on programming option-3 above) to quickly analyze an exported directory list for naming convention conflicts. I hope to also make it solve the issue by pre-creating the corrected path information on the destination in advance of recovery. -------- VERY IMPORTANT NOTE: The following will demonstrate the issue when Windows is not involved. Example; when using WinHex's "Export list ..." on a source file that is named "cross\platform.txt" (yes, that is the legal filename on the source), while creating an exported directory listing, WinHex faultily lists a folder called "cross" in the path column. That folder does not exist on the source. The resulting exported list looks like this:
There is no "\cross" folder on the source. The source path is really this:
During Recover/copy, objects contained in such paths are NOT recovered. The WinHex "Messages" windows lists them like before: 'Cannot create "z:\test folder\cross\platform.txt" ...' If Windows will not correct a cross platform naming convention violation, then WinHex really should. Windows is not at fault in this process, as it is not checking the exported list's content for naming usage. Even the WinHex Directory Browser displays this wrong in the path column, so I can safely say it is not Window's fault. Even more, the WinHex path display bar (below the tabs) will display this none-existent source folder under certain conditions! ya wanna know? BTW: Forensic users be aware; if you attempt use an exported list as evidence or a reference in court (especially where cross platform work is involved) you may want to run a validation script on that list first to ensure that it (or you) cannot be impeached by opposition council (they will use any 'minor' thing to their advantage!) If you are not aware of this 'fault', you may look comical and foolish under cross examination as you try to figure out the explanation for the discrepancies on the fly. Of course, none of us would ever be that unprepared or ignorant of our work, would we? (haha) --- I earlier stated that: >SR-7 still does not recover anything from within the source folder that has the NTFS objectionable name (i.e. under specific conditions ... the thread continued to: > Sorry, I don't have the time to further explain and defend against your wrong accusations ("it is the fault of WinHex that..."). I believe I have now demonstrated to my fullest that "under specific conditions" it is the fault of WinHex (even the DB has it wrong prior to Recover/copy). I could site more examples, I had some fun today with other OS's and FS's generating valid object names just to wreak havoc. However, I would rather see this stopped and fixed. Regardless of X-Ways position (defend status quo vs. fix), I have a budding fix that I can share. --- My goal is to try to eliminate as much work as possible, (e.g. potentially reviewing a multitude of needless messages generated during cross platform recovery and failed recoveries). If bad guys discover simple little things (e.g. add a trailing space or use a backslash) to evade, confound or slow down forensic recovery, they will do so. If opposition council finds an opportunity to impeach, they will do so. (Perhaps my next book will be "How to Evade Data Rcovery", it would probably sell well? haha) Thank you, Ross@WinPro.net | |||||||||||||||
| Ross Johnson Username: ross_winpro_net Registered: N/A |
Almost two years later ... WinHex 13.0 Recovery from an HFS+ many folders have trailing spaces (as allowed in HFS+) Windows will not create such a folder. WinHex "Messages" window lists way to many messages regarding: "Cannot write "c:\macrecovery \". Make sure the folder exists and the file is not write protected." Since, to the best of my knowledge, Windows still cannot make such a folder, it will never exist and I will have to spend many ours manually recovering the content of such folders. Wish: When WinHex encounters a trailing space in an HFS+ source, substitute it with an underscore! Please! Please! Thank you, Ross@WinPro.net | |||||||||||||||
| Stefan Fleischmann Username: admin Registered: 1-2001 |
> Windows will not create such a folder. I think it does, it merely changes its name by removing the training space. In v15.2 and later, WinHex and X-Ways Forensics will remove trailing spaces, too, OK? That way files in such directories can be recovered, too, because the actual path in the output directory as created by Windows on request of X-Ways Forensics will be expected by X-Ways Forensics. |