This is a major update of the OutWit Kernel.
Enhancement - Complete refactoring of all datasheets in the application (views and catch): they are now able to handle hundreds of thousands of rows.
Enhancement - Refactoring of the export functions to be able to handle very large volumes of data as well.
Enhancement - Fully compatible with Firefox 3.6 to 9.
Enhancement - More next page links are found, in more languages.
Enhancement - Scrapers were optimized, should be easier to produce and more forgiving.
Enhancement - When downloading files, an increment or a date was added to those with duplicate filenames. It is still the case, but when the url contains parameters, OutWit now adds these to the filename before testing for duplicates. This is more effective when the images come from databases.
Feature - Added a #nextPage# directive in scrapers, allowing you to tell OutWit Hub how to find the link to the next page in a series when it doesn't find it automatically. For the moment, this is only applied when the scraped view is active (bottom panel not at default settings).
Feature - Many new directives were also added to scrapers to help the debugging: #showSource#, #showMatches#, #showResults#, #showDelimiter#, #showScraperErrors#, #showServerErrors#, #simulate#, #showAlert#...
Feature - Added Lookup list replacement: #lookUp(value,listOfValuesToFind,listOfReplacementValues)# or #lookUp(value;listOfValuesToFind;listOfReplacementValues)# for replacing lists of values. The elements of the first list will be respectively replaced by those of the second.
Feature - Added new replacement functions: #(term1 operator term2)# and #if(condition,valueIfTrue,valueIfFalse)# or #if(condition;valueIfTrue;valueIfFalse)#. Works with the following operators: <,=,> (comparison operators); a=A (case-insensitive comparison); a==a (case-sensitive comparison); a!=b (not equal, case insensitive); a!==b (not equal, case sensitive); a+b (addition of integers: 1+3=4; concatenation of strings: out+wit=outwit; incrementing characters: c+3=f), a-b (subtraction of integers: 5-2=3 or decrementing chars: e-3=b ), a*b (multiplication), a/b (division) and a^b (power). The terms can be literals, variables or functions.
Feature - When using equality operators on strings (=, !=, ==, !==), you can now use the wildcard % in the second term to replace any string. (ex. these three statements are true: headstart = Head% ; homeland == h%d ; lighthouse = %HOUSE).
Feature - Added new replacement variables in scrapers: #URL#, #BASEURL#, #DOMAIN#.
Feature - Added the #nextToVisit(#myURL#)# function which, in the 'Replace' field, instructs the Hub to give the variable #myURL# the next scaped value which is not found in the list of visited URLs. This means that, used in conjunction with #nextPage# and #BACK# you can create complex scraping workflows. You can, in particular, create multi-level scraping processes.
Feature - Added the new directive #variable#myVariableName#. The occurrences of the variable (#myVariableName#) are replaced, at application time, by the scraped value in all other lines of the scraper.
Feature - Added URL alteration functions: #getParam(url,parameterName)# and #setParam(url,parameterName,parameterValue)#. When used with #URL# in the #nextPage# directive line, you can easily set the value of the next page url in some cases. ex.: #setParam(#URL#,page,#(#getParam(#URL#,page)#+1)#)# in the replacement field will generate the next url, incrementing the parameter 'page'.
Feature - The right-click menu on a scraper field in the scraper editor now allows you to highlight the matches in the source code. The same feature used on the description field, highlights matches for the whole scraper line.
Feature - Empty/Export/Download buttons were added to the datasheet bottom panels depending on the context, to export selected content.
Feature - It is now possible to completely disable images and plugins (like flash) in the Hub, for faster browsing: Use the right-click menu on 'page' in the side bar.
Feature - New settings were added to the time preferences, including temporization and pauses at set intervals for the fast scraping mode (XHR queries).
Feature - One major change is that scrapers can now reuse a same marker in several lines and use overlapping markers, which was not possible in the previous version.
Feature - Regular expressions can now be used in the find bar (ctrl-F or cmd-F) of the 'page', 'source' and 'scrapers' views! Just begin and end your patterns with "/" (i.e.: /yourRegularExpression/ ).
Feature - The right-click menu of the datasheets has changed: Automatic browsing and Scraper application are now gathered under "Auto-Explore Selected Links". In this submenu, you will find the exploration and scraping functions: 'Browse','Dig', Fast Scrape selected URLs (with the new possibility for the latter, to include the selected data fields in the scraping results) as well as the possibility to apply generic macros (i.e. macros that are not specific to a given URL) to the selected links.
Feature - You can now set the preferences so that FF runs OutWit Hub automatically on launch.
Fix - Corrected encoding problems in the Dynamic Source that could happen if the meta declaration was not UTF-8.
Fix - Corrected problem displaying some records with very large fields in the Detail panel.
Fix - Fast scraping works on very large selections of datasheet or catch rows.
Fix - Fixed the overwriting of existing files when manually saving export files.
Fix - Large number of fixes and performance optimizations throughout the code.
Fix - Several fixes in scrapers, in particular unwanted blank lines added in fast scraping mode.
Fix - The slideshow function now works even in on-demand images mode.