BPI inside

Tag: Computer/Programming

  • Saving an academic career or “How I secured data from a failed external HDD”

    Two weeks ago, a Friday afternoon, I received an emergency call from a friend. Her sister had stored all of her University work on an external harddrive. The rest of the story should be quite predictable from this moment on. Of course the data was stored nowhere else (this drive SHOULD have been only a backup drive but grew into the main working device) and suddenly one second to another the drive was not accessible anymore. Furthermore the drive emitted a clicking sound every few seconds. Several tries to get the drive running again failed so far. My friend already checked the most obvious causes like dirty contacts or loose connections and had no success so they gave me a call as they already apprehended a headcrash.

    When the drive arrived in my hands I had the following situation: an external, almost brand-new, flat-lying 1TB harddisk exposing USB and eSATA connectors. To lower the further damage during uptime a bit I immediately fixed it in a vertical position using a stand from another external disk I had lying around. In my imagination this should minimize the bouncing of particles some more (because some of them then collect on the lower side of the encasing) if the surface of the disks had indeed taken some damage and slow down the degradation to some degree. A very quick test from within Windows showed that the drive tried to register with the system but failed to do so. So no more tinkering here but quickly start up a Linux system for recovery.

    I started up SystemRescueCd which I had installed on an USB stick (using SARDU)for situations like those. Connecting the drive via eSATA failed because the drive didn’t show up in /dev/ so I had to fall back to the slower USB connection for all following steps. Connecting with USB took some time (about ~30-60s) until the device showed up in /dev but then it was relatively accessible. First thing I checked was the SMART info using

    smartctl -a /dev/sdd where it became pretty obvious that the drive is badly damaged. About 100 relocated sectors and a handful of pending relocations. Very strong signs for a headcrash indeed, so no time to waste and get as much data from the disk as possible.

    Trying to mount the disk failed so I could not just copy the files down but had to make a complete image at first to work with that later on without the failing drive. At this moment another problem struck as I had nothing around where I could store a 1TB image file. At maximum I could free up 600GiB on a Linux drive.

    I had to make another call to find out that there should be an NTFS filesystem on it with about 200GiB of data stored on it. The drive should be relatively new and there has been not a lot of activity beyond storing and some updating of the files. So I hoped for a lot of uninitialized areas which would be easily compressible. A quick check with

    hexedit /dev/sdd confirmed my speculation, there were large zeroed-out areas at the end of the disk. This confirmation took a while because I seemed to hit erronous areas already at the beginning of the disk where the tool stalled until the read-error timeout snapped it out.

    The Linux filesystem ext3 has support for sparse files which automatically compresses unused/zeroed areas of a file so I had the hope that the 1TB image-file would still fit on my 600GiB free space.

    A simple copy of /dev/sdd (with cp or dd) would fail because of the errors on the disk, luckily there are tools available which save the working areas and try to recover the failing areas. I chose ddrescue for this job because it has a buildin switch for creating sparse target images, which saved me from manually creating one. I somewhat sticked to the instructions from the Forensics wiki and made a first pass over the disk without retrying failing sectors to save as much of the intact data as possible.

    ddrescue -d -S -n /dev/sdd disk_sdd_sparse logfile

    This first run took quite some hours because transferring 1TB over USB at 30MB/sec (at best, almost zero when hitting defect sectors). Because of the logfile (the last parameter) I was able to interrupt the process overnight as I didn’t want to let it run unattended for too long. During the copy from time to time I checked the SMART infos in a second terminal which showed me that either the disk was dedgrading by the minute or the disk logic was just counting currently undetected errors. But the further the initial rescue was running, the larger were the intervals between the errors which raised my hopes. In the end the first run ended with the full 1TB image stored on my disk (which took only ~250GiB because of the Sparse option), having about 130MiB of errors scattered across ~1100 locations. Not that bad, but there was surely some more to gain, so on to the second run.

    In this second run I started ddrescue in a way where it looks closer to the erronous spots on the disk and tries to approximate to the exact location of the error within the whole error area to get out all bytes which are not really affected. These actions are called splitting and trimming of the defects.

    ddrescue -d -S /dev/sdd disk_sdd_sparse logfile

    This repair-run finished faster because it only checked the errors, nevertheless it still took some hours. It was quite successful as it lowered the number of error-locations to ~904 and the affected data area to 512kiB. Wow. I wonder if there’s more to squeeze out. Let’s retry the errors and automatically retry without retry-limit

    ddrescue -d -S --retrim --max-retries=-1 /dev/sdd disk_sdd_sparse logfile

    Again I let this run for some hours and when it seemed to only have minimal success anymore (about around the 5th automatic retry) it was down to 859 errors summing up to 490kiB of errors. So, finally the outcome of the rescue operation looked quite promising. Just for the curious ones, the smartctl statistics were far beyond good and evil with about 900 relocated sectors and 1300 pending. And big fat letters telling me "FAILING NOW"…

    The last step now was to mount the partition within the disk-image. I found out the offset for the partition mount by comparing the outputs of the following hexedits and finding the second one in the first one (luckily Linux could detect the partition itself).

    hexedit /dev/sdd hexedit /dev/sdd1

    If this weren’t possible I would have calculated the partition offset using one of the guides on the internet here (German) or here. After that I coult mount the partition using…

    mount disk_sdd_sparse /mnt/image -o ro,loop,offset=0x7e00

    … and began to copy the files out of the partition. There were some filename encoding issues and warnings during the copy which were finally resolved by mounting with a manually enforced charset.

    mount disk_sdd_sparse /mnt/image -o ro,loop,offset=0x7e00,iocharset=utf8

    Well, that’s the story of a saved academic career (at least a gigantic pile of work). I hope that my experiences maybe help someone other with rescuing data from a failing disk. Now I just have to decide what gift to take in exchange for this rescue operation… 😉

  • Easter Cleanup

    And another month went by without much news from me. There haven’t been many changes anyway. University is stressful as always, the amount of work at my employer keeps at a high level and there is only little spare time for relaxing.

    Furthermore over the last week, my laptop computer has been suffering a slow death wich I tried to prevent by repairing the failing operating system several times. Eventually I had to give up when the point came where the installation of some Windows updates rendered several subsystems (networking, graphics, …) unusable.

    I took the bitter pill and bought a new hardisk and installed a new OS from scratch. Currently I’m in the process of transferring all of my data from the old Hdd to the new Win7 installation. This will still take a few more days until I’m again at the point where I can effectively perform real work again but I’m slowly approaching that point.

    And what’s slowing this down too is the fact, that I’m sick since Friday and in bed since Saturday. Happy Easter, ha ha :/

  • Repartitioning relief

    Woke up, checked the computer, repartitioning finished. Total time: a bit over 10 hours.

    Whew….

  • Repartitioning worries

    This late afternoon I decided to do a bit of re-partitioning on my large external harddisk and make some more room on the FAT32-partition to make the data on it (mostly recorded broadcast videos) available for my LCD TV which only accepts FAT32 partitions.

    Everything went well in the beginning, the large 800GB partition was shrinked within two hours (yes, it takes a bit over USB) and the moving of the following partitions took only a few minutes more. But the enlargement of the FAT32 partition from 100 to 300GB is now already taking about six hours and I am getting a bit nervous…

    I’ll let it run overnight now and hope that it’ll be finished in the morning. But I don’t know how I should proceed, if it’s still busy when I get up.

  • Android development is just beginning

    Yesterday I spent a few hours trying to develop a minimalistic application for my [new android phone][1]. With the help of a quick [Hello World tutorial][2] I got up and running quite quickly. Within a few minutes I had Hello World showing up in the Android phone emulator.

    When I tried to start it on my real device, I had the problem, that my phone was not showing up as connected device when I plugged it in via USB but only its internal memory was available as drive. After some investigation and research I found out, that I plugged in my phone before I activated the USB debugging mode on the phone which causes some driver irritation on Windows. [This thread][3] explains the details and how to solve the issue.

    The next thing I did was to find out how to load the contacts stored on my SIM card. This is not a very well documented task so here is the required code to use Content Providers for loading contacts from the SIM module. This method returns the contacts as ArrayList of Strings which consist of the three fields name, number and _id.

    private ArrayList
    retrieveSIMContacts() { Uri uri15 = Uri.parse(“content://sim/adn/”); ContentResolver resolver = getContentResolver(); Cursor results = resolver.query(uri15, null, null, null, null); // Android 1.6 has a different URI if(null == results) { Uri uri16 = Uri.parse(“content://icc/adn/”); results = resolver.query(uri16, null, null, null, null); } final ArrayList simContacts = new ArrayList(); final int nameIndex = results.getColumnIndex(“name”); final int numberIndex = results.getColumnIndex(“number”); final int idIndex = results.getColumnIndex(“_id”); while (results.moveToNext()) { final StringBuilder builder = new StringBuilder(); builder.append(results.getString(nameIndex)).append(” : “); builder.append(results.getString(numberIndex)).append(” : “); builder.append(results.getInt(idIndex)); simContacts.add(builder.toString()); } return simContacts; } All in all I got a small app running on my real phone showing the contacts from both the phone itself and the SIM card in two listviews selectable via tabs within two or three hours. Quite fast, I’m impressed. I expected that Android development would be a bit more complicated… **Update 2010-01-04**: Update for Android 1.6 [1]: http://kosi2801.freepgs.com/2009/09/30/little_android.html “Little Android” [2]: http://www.ceveni.com/2009/06/writing-android-hello-world-program.html “Writing an Android Hello World Program” [3]: http://groups.google.com/group/android-developers/browse_thread/thread/48e92dcc0a8a9a23/bbff8436bfd23ce0?lnk=gst&q=G1%2FHTC+Magic+not+recognized+by+ADB+under+Windows#bbff8436bfd23ce0 “G1 & HTC Magic not recognized by ADB under Windows”

  • Java language pitfalls

    A recent blog entry posted in Argonauts blog deals with a C# codepiece which is valid at the first look and even compiles cleanly but failes gracefully with a runtime exception when executed.

    Argo shows the C# code for his example but (as he also mentions) the same code can be used in a Java example. The following class hirarchy…

    class BaseType {
}

class SubTypeA : BaseType {
}

class SubTypeA : BaseType {
}

    looks innocent so far. But if it is used the following way

    public static void main(String[] args) {
  BaseType[] baseArray = new SubTypeA[3];

  baseArray[0] = new BaseType();
  baseArray[1] = new SubTypeA();
  baseArray[2] = new SubTypeB();
}

    things get interesting. The code compiles without any problems (as in C#) but when executed, one is faced with an java.lang.ArrayStoreException. The cause for this is burried in the Java Language Specification 4.6 (thanks for the hint in the posts comments which saved me some searching). Upon compilation the type of the array is BaseType and after that it gets the SubTypeA-array assigned. The compiler does not know at this position that it would have to re-type the array it is assigning to to avoid the problems lying ahead. That’s why arrays are also checking their assigned objects at runtime and cause this exception if something invalid occours (as specified in JLS 4.7).

    I think that this problem would be solvable in Java, C# and most other languages which treat arrays the same way. But I also think that this would open a pandoras box ful of problems and complexity arising from this new constraint just for this specific issue. And personally I think this issue is not a common one and appears just in border-cases so that most developers can deal with it for now. Nevertheless I also think, this issue can be solved, it just may be too late for Java and C#.

  • Experiences with optimized Firefox builds

    I’m running Firefox on all my computers. And I have a habbit of collecting lots of tabs for later reading or further actions. Furthermore, a lot of the pages I’m visiting have at least a little bit of JavaScript running. Some of those use scripting quite extensively.

    So it was no surprise that over time the oldest of my computers at work, which I use for reading email, was getting slower and slower with Firefox using up to 0.5 GB of the available RAM. Which got paged out to the harddisk a lot, when I also switched back and forth to Outlook and some other applications. It became quite inconvenient to use.

    I decided to take action and try the speed- and memory-optimized Firefox-Builds from pigfoot (via Lifehacker). Installation consisted of unpacking the downloaded package (available as self-extracting archive or also as portable version) and copying over my Firefox-Installation (after making a backup of course).

    Upon startup the first thing I recognized was the new application icon and startup screen. And it felt just a bit faster, almost unnoticeable. In all other manners it behaved exactly like the original Firefox. But now after having it runnig continuously for some days, which would have made the old Firefox crawling like a disabled snail, it is still running fluently and reacting a lot faster than I expected it.

    I’m very satisfied with this build of Firefox and hope, that some of the optimizations somehow make it into the original build. I think many power-users like me would only appreciate that.

    One drawback should not stay unmentioned: I do not know, if this version follows the automatic updates from Mozilla. Nevertheless, new versions have to be installed manually after these are available at pigfoot’s weblog. But for the improvements I’m experiencing on my stressed computer I’m happy to live with that.

  • Fixing Cisco VPN Client BSOD on Windows XP

    Recently, after working some time without any problems, my Cisco VPN Client started again to cause bluescreens when connecting to a VPN server. This hiccup was already present when my mobile computer was brand new and I tried a lot of stuff but in the end the problem went away when I had to re-setup my computer for another reason.

    But now it had reappeared. The only thing I could think of which could have caused this was that I had installed a lot of updates for my HP mobile workstation and a lot of Windows updates.

    I then started again into the world wide web to find a solution while hoping that someone found a solution to my problem in the meantime. And indeed, this posting solved my problem. It works by disabling the (hidden) vsdatant device in the device manager.

    1. open the device manager
    2. select devices by connection from the view menu
    3. select show hidden devices, again from the view menu
    4. now in the devices list scroll down to the bottom until you find the vsdatant entry
    5. right-click and disable the device
    6. you’ll be presented with a reboot-request, which you should accept before trying to connect to a VPN now

    I do not know exactly what this vsdatant device is for but from hints in the internet it seems to be connected to the TrueVector (or stateful) firewall, which is included in the Cisco VPN Client. As I do not need this (or know someone who does) and furthermore found no articles in the net indicating further problems with a deactivated vsdatant I consider the case solved for me now.

  • JBoss Web Service issues with JDK6

    The past two weeks I’ve working on an exercise for the university. The task was to create [Enterprise Java Beans][1] (EJB) and [Web Services][2] (WS) which were then deployed to and run on the [JBoss Application Server][3]. We were using Eclipse Ganymede EE, Java JDK6 and JBoss version 5.0.1 GA for our purposes.

    My colleagues and I had quite some troubles getting it all to work but eventually we managed it because most of the issues are documented and their resolutions are available on several sites somewhere on the net. Only this particular issue seemed to be covered nowhere or if it was, the solutions suggested did not work for us. I’m giving you the solution that worked for us here, maybe it helps somebody who runs into the same issues we’ve experienced.

    Many people seem to succeed with the [Instructions for using JBoss5 on Java6][6] from the JBoss Getting Started Guide, but for whatever reason this did not work for us. If you have the same problem persisting even after copying around the library-files, read on.

    I hope the solution present here can also help you with your problem. If you have anything to add, find a mistake or can give any other feedback, please leave me a comment.

    The issue

    JBoss is up and running without trouble, having webservices deployed and visible on the webservice-endpoint overview at http://localhost:8080/jbossws/services. But upon calling one of the methods provided the JBoss chokes with the exception:

    ERROR [SOAPFaultHelperJAXWS] SOAP request exception
java.lang.UnsupportedOperationException: setProperty must be overridden by all subclasses of SOAPMessage
    at javax.xml.soap.SOAPMessage.setProperty(SOAPMessage.java:445)
    at org.jboss.ws.core.soap.SOAPMessageImpl.
    (SOAPMessageImpl.java:82) at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:215) … (and some more exceptions with “setProperty must be overriden…” appearing several times) and the webservice call fails with no result. ## Quickly explained solution 1. install JDK5 and add it to Eclipses installed Runtime Environments 2. set JBoss execution environment to use JRE5 3. set JRE in webservice-projects buildpath to JRE5 4. set Project Facet in webservice-project for Java to 5.0 ## Detailed solution 1. [download JDK5][4] from Sun 2. install JDK5 (only JDK needed, no seperate JRE, docs, etc.) 3. add JDK5 to Eclipse as Installed JRE (Window->Preferences->Java->Installed JREs->Search…) 4. change startup configuration of JBoss to JDK5 (doublecklick on JBoss-entry in Servers-tab->General Information->Open launch configuration->JRE->Alternate JRE) 5. change the projects JRE System Library to use JDK5. Either via right-click on the system library entry in the project->Properties or right-click on the project->Build Path->Configure Build Path->select “JRE System Library”->Edit. Then set the Alternate JRE to JDK5 6. change the projects Java facet to JDK5 (right click on project->Properties->Project Facets->Change the setting next to the “Java” entry to “5.0” 7. check, that in %JBOSS_PATH%/server/default/deploy (or your own configured deploy-path in JBoss) there are no .war-files left from your current project 8. if not already suggested by Eclipse itself, rebuild the webservice project (if building automatically just clean the project via Project->Clean…) 9. start JBoss via Eclipse 10. check that somewhere near the beginning of the JBoss console log something like “Java Version: 1.5.xxx Sun Microsystems Inc” or “Java HotSpot(TM) Client VM 1.5.xxx” appears as “INFO [ServerInfo]” log line If you are receiving errors like “java.lang.UnsupportedClassVersionError: Bad version number in .class file” then there is still something connected with JDK6 left in your project. Check the steps again and also your included libraries if there is something suspicious and then rebuild the project. After all these steps you should finally be able to call web service methods on your endpoint without causing those exception anymore. ## Issue background The technical details and origins of this error are explained in [JBWS-2649][5] along with an [initial solution][6]. In short JRE/JDK6 includes a dummy-implementation for this setProperty() method which overrides the required implementation which is supplied with JBoss. The mentioned initial solution solves the problem by copying the supplied libraries to a location in the classpath where it should be loaded before the JRE libraries but as already mentioned this did not work for most of our class. [1]: http://java.sun.com/products/ejb/ “Enterprise JavaBeans Technology” [2]: http://java.sun.com/javase/technologies/webservices.jsp “Web Services” [3]: http://www.jboss.org/jbossas/ “JBoss Application Server” [4]: http://java.sun.com/javase/downloads/5/jdk “java.sun.com – Download SE Development Kit 5.0” [5]: https://jira.jboss.org/jira/browse/JBWS-2649 “JBWS-2649: SOAPMessage implementation bug” [6]: http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Installation_And_Getting_Started_Guide/5/html_single/index.html#Java6_Notes “Installation and getting started with JBoss on JDK6”

  • Upcoming open source project

    For one of my lectures at the university we had to process a project. I chose a software project with a topic I suggested myself and possibly could also later use at work.

    Meanwhile my project has grown to a not-so-bad framework and I’m really thinking of opening it to the public after it has been completed at the university.

    I already checked it with the university I just have to talk to some people at work on that issue.

    So expect a new java project on SourceForge.net in the next weeks.