Monday, 15 April 2013

Fast DLI Downloader Tool to download PDF Books from Digital Library Of India

Digital Library Of India [http://www.dli.ernet.in/] hosts millions of copyright freed scanned books in English and Indian Languages for public access. Unfortunately this website does not provide any mechanism to download the entire e-book in any format. I have created an open source Java project to allow people download the same.

Key Features of DLI downloader

  1. Search and Download Books right within the DLI Downloader tool.
  2. Queue up your download items in the tool and let it download books silently for you.
  3. Both TIF and PDF version of the e-book can be saved on hard drive.
  4. Application can minimize to system tray so as not to interfere with your other tasks & reducing the memory footprint.
  5. If you shutdown the application gracefully then the un-finished downloads will become active upon the next launch of the application.
  6. Configure the download speed of the application over a metered network so that the tool does not eat up all the available bandwidth of the network.
  7. Bulk download by putting all required barcodes into a text file and enter the path for text file into the tool.
  8. Automatic error handling for the corrupt file downloads due to unknown errors at the time of PDF conversion.
  9. High quality TIFF to PDF conversion which can be configured in the settings.
  10. Network tolerance can be configured for working with slow network bandwidth.
  11. Barcode related metadata is stamped on the PDF on its first page and in the file properties. The information never gets lost.
  12. Works well on x64 & x86 on Windows 7, Windows 8, Mac And Ubuntu 12.10.
  13. Above all, everything for Free !!

How it Works ?

Suppose you are looking for Chikitsa Books on DLI, just enter the word chikitsa in the Search Tab and results will appear. If a book is already downloaded by you then it will be shown in green color mentioning local. If you want to know more about the search query syntax then look for the following document.
http://lucene.apache.org/core/old_versioned_docs/versions/2_9_1/queryparsersyntax.html

Searching for the Desired Book

Now you can select the book of your choice and right click on it to download it to the desired location.

Downloading the Book in PDF format

Sometimes we want to configure settings for the dli downloader, the Settings Tabs is right for this work. Don't change the settings if you do not understand it.

Configuring the Settings

System Requirements

  1. Java 8 runtime environment (http://java.com/en/download/index.jsp)
  2. Windows 7, Windows 8, Mac, Ubuntu 12.10 x64 & x86
  3. Latest DLI Downloader release from https://github.com/cancerian0684/dli-downloader/releases

How To Run ?

  1. Run the dli-downloader jar by double clicking it on windows. Unix and Mac users please do run the following command
    java -jar dli-downloader-5.9-jar-with-dependencies
  2. Search the desired book in Search Tab and right click to download the book.
  3. Your download will start or queue up.
  4. Once you are done with all the downloads, then find the DLI icon in the system tray and right click on it, then choose exit to quit the application.
  5. That's it, enjoy !!!

Configuration Settings Explained

  • height - height of he application windows. Any changes in the gui are automatically saved in the XML
  • width - width of the application window.
  • x & y - are the screen locations of the DLI Window.
  • numberParallelJobs - number of parallel downloads at any given point in time.
  • readTimeOutMs - this is socket read time out value before throwing an timeout exception in milli seconds.
  • rootDirectory - directory where the fresh downloads will be saved.
  • speedLimitKBps - maximum speed limit that the application will use for all its downloads.
  • threadsPerJob - number of threads that will download pages for a given barcode.
  • deleteTifIfSuccessful - if set to true then the TIFF files will be deleted if PDF conversion and tiff download goes successful.
  • maxRetryCount- If connection to server is very slow then the connection timeout exception can occur. In case the download fails, then this property will decide how many times to retry the failed download.
  • maxConsecutiveFailure - maximum number of consecutive IOExceptions from the server before the tool stops downloading the given barcode.
  • downloadDirectories- comma or semicolon separated list of directories which can contain the already downloaded PDF files. The pdf file must contain the barcode in its name. This directory will be scanned by the tool at the time of startup for caching the existing downloads and a warning prompt will appear to user if he tries to download the same barcode again.
  • quality - TIFF to PDF conversion quality. Maximum quality is A2, minimum is A7
  • lookAndFeelNumber - 0 is SystemLookAndFeel, 1 is NimbusLookAndFeel, 2 is WindowsLookAndFeel
  • createBarcodePage - setting it to true will create an initial page displaying the metadata of the barcode.

Grabbing the Source Code

This utility is written entirely in Core Java & Swings framework. An open source project is hosted at https://github.com/cancerian0684/dli-downloader/
Feel free to download the source / binaries from the Google project hosting


Java Source Key Features 

The Java Implementation of this tool embeds various design patterns & Algorithms like,
  1. Token Bucket Algorithm for managing the download speed
  2. Observer Design pattern which enables Swings GUI to be observer of the running download task
  3. Concurrency Utilities like Executor Framework has been utilized for the parallel processing and maximized throughput.

Feedback

For any feedback or suggestions, write back to me cancerian0684@gmail.com or leave a comment on this webpage.


Donation

If anyone feel like donating money for this work, feel free to do that. I will use that money for charity work (Tree Plantation & Gow Palan, etc)

Bank Deposit by NEFT

MUNISH CHANDEL
a/c 5277618224
IFSC : CITI0000002
Citi Bank N.A. Delhi

Don't forget to send me the details on email : cancerian0684@gmail.com

56 comments:

  1. i gone through your website its really good information, we are also using much similar PRODUCTS, You can visit us.

    BARCODE INIDA



    .

    ReplyDelete
  2. Munish,

    Your software has fulfilled my long cherished wish. I have been using IDM till now for downloading from DLI. I downloaded and tried your DLI downloader just now. It is pretty fast compared to previously available tools. Another good feature is that one can add any number of books for download as long as internet connection is fast enough.

    Only drawback of this tool is that page size is changed to A4 size. But, this problem is negligible compared to the good features of the software.

    Thank you very much for developing this wonderful software.

    ReplyDelete
  3. Hi Shankara, thanks for using this software and providing the right feedback. I have changed to TIFF to PDF quality to A2 (that is the scanning quality of DLI scanner, I believe), Please use Version 2.6 to avail this change
    http://code.google.com/p/dli-downloader/downloads/list

    ReplyDelete
  4. dear manish,

    can i use this downloader in MAc machine? Please iform me. shrivara@gmail.com

    ReplyDelete
    Replies
    1. after installing Java 7 on Mac, you can run the following command on the command prompt

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      that should work

      Delete
    2. And please do let me know if it works on Mac or not. I am keen to know that !!

      Delete
    3. Hi SamanA,
      We have now tested it on Mac as well and it works just fine. So just go ahead and enjoy the software on your macbook :)
      P.S. : Due to security restrictions, Mac wont open the file the first time try to by double clicking it. You will have to right click and select Open With option. Next time onwards, it will open with double click as well.

      Delete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. Munish,

    You have really done a great job ! The multiple download feature is the most desireable one for all DLI lovers. Also, you have changed the page size from A4 to A2. I was using IDM previously, but as you have added the multiple download feature, I am very happy to use your downloader. Thank you very very much !

    ReplyDelete
  7. Hi Shankara, I have rolled out new version of software accommodating few new changes -
    1.) Remove all option on the GUI
    2.) Configurable PDF Quality (A2 to A7) in dli-settings.xml, automatic detection is tough to implement because few barcodes have different sized TIFFS for different pages. A2 is the maximum quality, A7 is minimum.

    ReplyDelete
  8. Hi Munish

    I have a requirement. I can download all the tiff files for a given barcode number to a folder. Now converting that tiff files to Pdf can I use your classes to do it. If this is possible can you please help me to know how to do that in a simple java file. That would be a much helpful for me.

    Thanks
    C.R.Bala
    Bangalore

    ReplyDelete
    Replies
    1. There are two ways to do it -
      1.)
      a) Place all the TIFF files in a folder and name this folder as the barcode i.e. 5990010100576
      b) Now start this tool and configure the suitable download location i.e. E:/downloads
      c) then cut and paste your barcode folder into this location.
      d) now enter this barcode into tool's download list, tool will scan the metadata from web and then try to download the TIFF files into barcode folder. But seeing the already existing tiff files, it will not download them again. Tool will just reuse your tiff files and convert them to PDF

      I have done this many times and should definitely work for you.

      2) You can grab the source code for TiffToPDF.java from
      https://code.google.com/p/dli-downloader/source/checkout and try importing project into inellij/etc
      This file does the conversion using iText as the underlying library.

      3) Just check out how iText libray works, its very simple to convert tiff to pdf using iText

      Delete
  9. How about Linux users sir, whether the same tool works? Or do we have a Linux version too? Please clarify and provide link for the Linux version as needed. Thanks.

    ReplyDelete
    Replies
    1. I have tested this tool in Ubuntu 12.10 x64 and it works perfectly fine (though i tested few versions of it only). The only thing that you need is Java 7 installed on ubuntu. Download the jar from https://code.google.com/p/dli-downloader/downloads/list
      and then run the following command

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      Delete
  10. Thanks for the tool Munish.
    I had trouble downloading initially it used to fetch book details and start the download and change the status to Failed immediately.
    Then i realized i am not running latest version of Java. And after updating the java it is working absolutely fine.

    ReplyDelete
    Replies
    1. great, In case you are a developer and understand Java, then you can always run the tool from command prompt using following command

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      This will print every exception details on the command prompt making debugging/ troubleshooting easy.

      Delete
  11. Thanks for creating a nice tool for downloading books from DLI.
    Although, this tool worked well for me, I'm now seeing this in terminal :
    D:\DLI>java -jar DLIDownloader-3.0-jar-with-dependencies.jar
    [AWT-EventQueue-0] INFO org.shunya.dli.Utils - Loading File : dli-settings.xml
    [AWT-EventQueue-0] INFO org.shunya.dli.Utils - Loading File : dli-jobs.xml
    Could not load XML file : dli-jobs.xml
    javax.xml.bind.UnmarshalException
    - with linked exception:
    [java.io.FileNotFoundException: D:\DLI\dli-jobs.xml (The system cannot find the
    file specified)]
    I'm on Windows 8. I'm using DLI downloader 3.0.
    When I enter barcode and press 'Add', I can't see anything added. I did it multiple times. I've tried shutting application down and then running again. I've deleted old versions and xmls. Nothing seems to work.
    Although, I'm now updating Java in hope to get it working.
    Any suggestions ?

    ReplyDelete
    Replies
    1. Hi Dear,
      The exception that you are getting on console is normal (UnmarshalException) and will happen just for the first time when their is no job file created. I wonder why it is not picking up the job, because i have tested version 3.0 on windows 8 itself and it works just great. Even Java version should not be a problem as long as it is Java 7. Can you provide me some more information on my email : cancerian0684@gmail.com

      1) java -version
      2) does DLI 2.9 version work on your system ? https://dli-downloader.googlecode.com/files/DLIDownloader-2.9-jar-with-dependencies.jar
      3) Which barcode are you trying to add ?
      4) does the screen remain empty even after adding barcode ?
      5) can you put the barcode into a text file and put the filepath of that text file into tool.
      6) Can you send the complete logs from console after you try adding a barcode into the tool ?

      Delete
    2. Please do update if it worked well for you or not ?

      Delete
    3. 1. java-version : 7.21
      2. It works with same problems persisting.
      3. Anonymous barocdes were used.
      4. Yes.
      5. Putting barcodes in job.xml file works for me.
      6. The previously sent log was complete.

      Delete
    4. It seems like there is some configuration on your system which is not respected by the tool and it fails. I will try to create a new build for getting more clues for the same. Will update you once ready.. that build will just help me finding the real problem that is happening.

      Delete
    5. Thanks for looking after the problem.
      Although, I can assure you that I've this software working fine and without changing anything in my System Settings or Java Settings, it started behaving like that.

      Delete
  12. Thank you very much for this great and useful tool, Munish!

    One question: when the PDF gets created, it produses a barcode page as its first page. Is there an option to leave this out?

    Thanks again for this great job!

    ReplyDelete
    Replies
    1. Hey you can download version 3.1 from
      https://code.google.com/p/dli-downloader/downloads/list

      Right click on the system tray icon for DLI and choose settings.

      Choose this property and set it to false (by double clicking on it)
      createBarcodePage false

      this does not require a tool restart, but make sure you shutdown the tool using shutdown option otherwise it will not save the settings upon exit.

      This should solve your problem.

      Delete
    2. Thank you very much for your concern. It works perfectly!
      Best wishes.

      Delete
  13. When I shutdown the downloader from notification area, it just saves running barcodes in job.xml. queued barcodes are not saved.

    Another problem I'm facing is: if internet connection drops and resumes, you will have many corrupt .tiff files. DLI downloader considers them complete and skips to download left part. It will be better if it could match size of tiff to that on server.
    When such corrupt files are situated in tiff directory, DLI downloader could not make pdf - is also observed.

    ReplyDelete
    Replies
    1. I will look for the reasons why tool does not save the queued items at the time of shutdown.
      Regarding partial download of the TIFF files in intermittent connections, many times it is not possible to get the file size from the server to match against. But in TIFF to PDF conversion when such partially downloaded files fail to convert, tool retry once more to re-download the TIFF file and do the conversion once again. So Ideally even if there are partial downloads, PDF conversion should bridge the gap and re-download the files if there is internet connection available at that time.

      Delete
    2. Thanks for considering issues presented by me.
      When creating PDF from partially downloaded tiff files, the DLI downloader gives me 'FAILED' error.

      Delete
    3. when you get "FAILED" error for partially downloaded file, right click on the task and select "View Logs". Then copy all the contents and email to me cancerian0684@gmail.com

      Without logs, it would be hard for me to comment anything.

      Delete
    4. Whenever something fails, a log file is created in the download directory with name .log
      Please send that to me over the email.

      Delete
    5. With version 3.4 my problem of failed conversion is solved. Partial downloads are completed by software before completing conversion.
      This version also saves my queued tasks well.
      Although, I observed that right clicking notification icon and shutting it doesn't shut it always. A click on that icon brings the software back. This is unexpected.
      One more thing, right clicking running task and stopping it is causing hang.
      Thanks for solving my previous problems.

      Delete
    6. Shutdown takes a lot of time because it tries to gracefully Stop all the running Task, if possible so that no file is left in corrupt state. It is little tricky to handle quicker shutdown but I have given a try to handle it in slightly better way. Can you please check it with version 3.5
      The same change should solve the stopping task hang problem.
      I would say the hang problem is definitely solved, but shutdown problem should be slightly improved.

      Delete
  14. I don't know whether others have noted it or not, but I've seen that in some cases the actual number of pages on server is more than the listed one.
    I found it when using another method of downloading DLI books.
    If possible make sure that DLI Downloader does check that.

    ReplyDelete
    Replies
    1. Earlier I was just checking for the TotalPages, now I have added functionality to check for the extra pages in the end. But I have no such barcode to test, can you help testing the same in version 3.5

      Delete
  15. Munishji,

    Downloaded and tried 4.1 version. It has some wonderful new features, especially the search facility. "Remove all completed" is also a good feature. Thanks a lot for devoting your valuable time for great tool.

    ReplyDelete
  16. Tried to download tamil books, but search is not listing the same. Pls help

    ReplyDelete
    Replies
    1. Please download version 4.2 from http://code.google.com/p/dli-downloader/downloads/list

      Goto Downloads Tab and right click on the panel. Select "Index Language" option and then type Tamil in there. Wait for the indexing to complete and search should work fine after that. Please do let me know if that works for you.

      Delete
    2. Munisji,

      Thank you. Did as per your advice. downloads working fine. thank you once again.

      Delete
  17. Hi Munish,
    Thanks for this program. I was vexed with the other DLI downloader, which is very faulty.
    I liked this program very much. I was so much amazed by this small program that I wanted to explore the source code myself. I used the git command
    git clone https://code.google.com/p/dli-downloader/
    as provided int the site but i get an error saying that the repository is empty.
    can you let me know how to download source for this program?
    Once again thanks for this program.

    ReplyDelete
    Replies
    1. you can checkout the source from below location -
      https://github.com/cancerian0684/dli-downloader/

      Delete
  18. hi,
    i like this so much after reading but having error
    "there is problem with internet conectivity"

    my system uses a proxy to connect to internet
    please help

    ReplyDelete
  19. Do you know why I may be getting this on my mac (10.6.8) -

    $ java -jar dli-downloader-5.4-jar-with-dependencies.jar
    Exception in thread "main" java.lang.UnsupportedClassVersionError: org/shunya/dli/Main : Unsupported major.minor version 51.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

    ReplyDelete
  20. Hello Munish Cahandal, Your downloader is amazing compared to everything else out there. Keep up the good work

    ReplyDelete
  21. Sir, How do I remove an item from the downloads list? Where is the configuration file? I use 5.4 version in MAC OSX. I am not getting any context menu with right clicking.

    ReplyDelete
    Replies
    1. Dear Prasad, I suggest to upgrade the DLI downloader tool to version 5.9 (running/ pending jobs will not migrate) to get many other benefits of the newer release. You need to upgrade the java to Java 8 before installing this new version from here - https://github.com/cancerian0684/dli-downloader/releases/tag/5.9

      Delete
  22. hi there,
    i dont understand java unfortunately and i cant figure out what to do with the .JAR file after downloading it? do i have to unzip it using winrar software??
    could u please help me?
    by the way , i have installed java8 and i use windows 7 [32 bit]

    ReplyDelete
    Replies
    1. If you have Java 8 installed, then double clicking the downloaded jar file should launch the application. If that does not happen then select the downloaded file, right click -> open with -> \bin\javaw.exe where is something like C:\Program Files\Java\jre1.8.0_20

      Delete
  23. Hi Munish, which version will work on my samsung lg gpad? Thanks.

    ReplyDelete
  24. Hi Munish, which version will work on my samsung lg gpad? Thanks.

    ReplyDelete
  25. can't download books in .pdf format from http://www.new.dli.ernet.in/ , how can I do it, with or without DIL-downloader?

    ReplyDelete
  26. A lot of thanks for making such a wonderful down-loader for book lovers.
    I am facing a problem while putting Bar-codes from "dli.ernet.in" the down-loader does not respond.
    I am using 5.4 jar.please provide a solution.

    ReplyDelete
    Replies
    1. Can you please try with the latest version of DLI (requires Java 8) from this location - http://dli.shunyafoundation.com/

      Delete
  27. Right clicking does not show option of index language. Index cannot be updated.

    ReplyDelete
  28. Once the torrent has been created, uTorrent will ask where you would like to save the file. Try to make the description as short yet as descriptive as possible. Torrent Safely

    ReplyDelete
  29. why now a days dli sanskrit document page won't allow us to download tiff book papers.

    ReplyDelete
  30. Why people still use to read news papers when in this technological globe all is accessible on net? aol email login

    ReplyDelete

 

Sample text

Sample Text

Sample Text