Multiple Source Xmltv Downloader., SafeXMLTV Options

[Calvi]

Multiple Source XMLTV Downloader.

This is a project I have developed to improve the reliability of EPG data.
The package creates a new guide using the best items from multiple sources.
It does this based on the data in the Webscheduler Matchlist first, then prioritises by
SubTitle and Length of Description building a composite guide containing the most
possible matchlist matches, most sub-titles and longest descriptions given a range
of source guides.
Also sets repeat flags based on previous recordings.

The complete package is available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV.7z

Version History......

Download vbs only from......
(smaller download if this is all thats changed).
http://mail.lewis.com.au/public/xmltv/xmltv.7z

CODE

10.6 (Full Package)
  *Fixed bug with special characters in matchlists and repeat items.
  *Removed timezone detection from default configuration. (causes more trouble than it fixes)
  *Removed TTC from default configuration. (epgscan no longer in caps for Nine and Seven)
  *Updated to EPGScan 1.0.3.3  

10.5 (xmltv.bat & config.xml)
  *Updated to "7.30" from "The 7.30 Report" in <tzdetect>
  *Updated EPGScan to 1.0.2.0  
  *Fixed bug in xmltv.bat (thanks MarkG)  

10.4 (xmltv.vbs only*)
  *Modified the Remove overlaps code to use the same priority methods,
    ie. the priority of SubTitle over Empty, Match etc is preserved during the overlap removal.
  *XML Output is now CR LF instead of LF only.
  *Detect and remove duplicate items in each output program, eg if there are two or more
    <category>News</category> then the duplicates are removed.
  *Removed duplicate line regex from config.xml* as its superseded by above
    ^(.*)(\r?\n\1)+$::$1    
  *Added empty line between programmes for legibility.
  *Improved some logging to make more sense

10.3 (xmltv.vbs only)
  *Fixed bug in handling xml files with text comments embedded in programmes.
    http://forums.dvbowners.com/index.php?s=&showtopic=6598&view=findpost&p=79240

10.2 (complete package)
  *Added Channel Eleven (Melbourne Only)
  *Changed <filter> to Pre Replace Feature
    <prereplace> = a list of items to be replaced for data consistency or other needs BEFORE each source is looked at.
               this is a regex expression containing characters to be filtered out. eg.
               [\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\xa0] aggressive filter
               [\x00-\x08\x0b\x0c\x0e-\x1f\x7f]      less aggressive
               Also can strip /replace words etc. Eg. movie: for IceTV.
               leave blank to apply no filtering.
                Note: This filter is run per source prior to matchlist, merging checks etc.
                Eg. The following example removes invalid characters and Movie:, (New) and -Series Premiere from the title.
                  <![CDATA[
                  [\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\xa0]::;;
                  <title.*>.*?movie.?: ?(.*)?<(?i)::<title>$1<;;
                  <title.*>(.*) ?\(new.*\).*<(?i)::<title>$1<;;
                  <title.*>(.*) - series premiere<(?i)::<title>$1<
                  ]]>        
  *Added Post Replace Feature
    <postreplace> = a list of items to be replaced for data consistency or other needs AFTER guide is compiled.
                The list comprises delimited strings ("::") separated by a (";;") with the following information.
                Search Pattern::Replacement Data
                Search Pattern   - A regex expression to search for within each programme.
                Replacement Data - The data to replace the search pattern with.
                  Note: Use Cdata to escape xml tags, otherwise escape as per xml eg, < = &lt; etc.
                  Eg. The following example replaces Categories, Comedies with Comedy and Anime or Cartoon with Animation.
                    <![CDATA[
                    <category.*>Comedies<(?i)::<category>Comedy<;;
                    <category.*>(Anime|Cartoon)<(?i)::<category>Animation<
                    ]]>
                  Note: An incorrect regex or replace term can make the xml unreadable.
                        SafeXMLTV will check each element for validity before adding to the final guide,
                        but if corrupted it will be removed from the epg thus programmes will be LOST!
                        Corrupted programmes are logged.  
  *Added Repeat Tag Config Item.
    <repeattag> = The configuration for setting a programme as a repeat.
                  This item consists of a delimited string ("::") with the following information.
                  Insertion Point::Repeat Tag Data
                  Insertion Point - The text to search for within each programme to place the repeat data.
                  Repeat Tag Data - The data to insert for a repeat programme.
                    Note: Use Cdata to escape xml tags, otherwise escape as per xml eg, < = &lt; etc.
                    Default: The default inserts before the </programme> tag (ie last) with <previously-shown></previously-shown>
                      <<![CDATA[
                      </programme>::<previously-shown></previously-shown></programme>
                      ]]>
                    Eg. An Alternate Example sets the first Category instead (the trailing > is omitted to cater for lang="xx" attributes)...
                      <<![CDATA[
                      <category::<category>previously-captured</category>
                      ]]>
  *Added per programme element xml checking for validity before writing the final xml file.
     This mainly to address the possibility of damaging the xml file with regex replacements                      
  *Added a setting to enable/disable credits appended to the description.
    <dcredits> = Whether to append credits to the description tag.
                 Note: Credits already in the description are omitted and those missing are added with Additional Credits:
                       If no credits exist in the description then Credits: is appended to the end of the description.
                       If dcredits is set to false then no credits are ever added to the description.
  *Refactored a lot of code for performance improvements (~100% faster).                      
  *Other small improvements.

10.1 (xmltv.vbs only)
  *Fixed bug in repeatslist http://forums.dvbowners.com/index.php?s=&showtopic=6598&view=findpost&p=79081 thanks Ratall.

10.0 (complete package)
  *Added overrunners feature.
    <overrunners> = a list of shows that often overrun used to help add time to matchlist items that follow them.
    The list comprises delimited strings ("::") separated by a (";;") with the following information.
      Show Title::Add Time::Channel::Queue Depth
      Show Title - The name of a show that often overruns its alotted time as a regular expression.
      Add Time - The time to add to the following matchlist item in minutes. Typically the time this show overruns by.
      Channel - The channel the show is on as a regular expression (^.* matches all channels).
        Note:(Use the Output mapped channel)
      Queue Depth - The number of shows to look forward from an overrunner. Eg 2 means to look 2 shows forward only.
  *<exclude> now uses regular expressions.
  *<repeatexclude> now uses regular expressions.
  *<tzdetect> now uses regular expressions.
  *<sources> now include retries parameter. This allows timeout to be per grabber attempt and not handled by bat files.      
  *Complete rewrite of logging code
    Creates a full debug text log and a html email log in colour for better legibility.
    Title in green, summary blue, warnings orange and errors in red.
    Lists channels missing data.  
    Logs moved to separate subdirectory with rotation.
      <logdays> = the number of days to keep log files.
      <logroll> = the number of hours before rolling over the log files.
  *Rotation Code now common for xml files and logs.    
  *Added Channel 31 (C31) and ABC News (Still called ABC HD).
  *Added Channelmap settings for 7Mate (Still called Seven HD).
  *Added Channelmap settings for GEM (Still called Nine HD).    
  *Changed abcjc to allow for new abc website locations.
  *Added cache cleanup to sbsgrabber script.
  *Updated EPGSCAN to 1.0.0.9 (1.0.0.10 loops through all cards for me even if I don't want it to)        
  *All code reformatted to 80 columns.
  *Remove any hard coded directories to allow installation anywhere, esp for 64 bit etc.
  *Altered EPGScan timeouts in config as terminating tasks undesireable.

Version 9 no longer supported.

Archive of Version 9 available from.......
http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V9.zip

CODE

9.5 (xmltv.vbs only)
  *Added ability to not specify a repeat file in config. (Would error otherwise).

9.4 (complete package)
  *Added checks for channels missing data in each source
  *Log the channels with data vs mapped channels
  *Fixed a couple of channel maps where source names had changed.

9.3 (xmltv.vbs & config.xml)
  *Added an exclude option for the repeat database. Allows Subtitles like "All New" to be excluded from being flagged as a repeat.
  *Added ABC3 for Melbourne Location.

9.2 (xmltv.vbs only)
  *Improved performance by converting main dictionary to sorted list. (Dictionary Sort no longer needed.)

9.1 (xmltv.vbs only)
  *Improved performance of non alphanumeric character stripping
  *Strip non alphanumeric characters from existing repeat database, helps if manually adding.  

9.0 (complete package)
  *Added repeat feature. This feature will scan the TV scheduler XML capture directory
   and search for the same title:subtitle pairs in the guide. If found it will add a
   <previously-shown> element to the program in the guide. This can be used with the
   repeat flag in TV Scheduler to prevent repeat recordings of the same episodes etc.  
   .Net is required for this feature as a sorted.list is used. (Faster than dictionary object + sort)
    <repeatfile> = the name or the repeatlist file (A text file containing # delimited
    title#subtitle pairs as records. Note: ~ indicates a potential merged recording.)
    <captures> = the location of Web/TV Scheduler capture xml files (Note: subfolders are recursed)  
  *Added Channel 7TWO (Seven2) Note: Melbourne Only.
  *Misc changes & improvements.

Version 8 no longer supported.

Archive of Version 8 available from.......
http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V8.zip

CODE

8.6 (xmltv.vbs)
  *Improved Case Matching for Matchlists. Now accepts (?i) For regex items & obeys case settings.

8.5 (complete package)
  *Added Timezone Lookahead parameter and better logging to prevent tz changes a week early.
  *Added Richard Palmers Sydney Config Files.
  *Removed Chinese news from timezone correction (moves around too much).

8.4 (xmltv.vbs & abcjc.vbs & channelmap.xml for Perth EPGScan Nine)
  *Made modifications to abcjc grabber to deal with #$%&*@! ABC website out of order.
  *Changed Perth EPGScan GO! PID to 1026  

8.3 (complete package)
  *Added new tvscheduler pro directories.
  *Added Channel Go to melbourne config.
  *Added Overlap and gap repair options (use with caution!)
    <repmaxolapmin> = the maximum minutes an overlap may be to repair
      0 = disabled
      10 = default - any overlaps 10 minutes or less will be corrected.
      Note: Not active unless <remmaxolappct> is also active
      Note: This makes a tidier guide but can shorten programs incorrectly.
      Note: matchlist matches are never shortened.
    <repmaxolappct> = the maximum percentage an overlap may be to repair.
      0 = disabled
      25 = default - any overlaps 25% of duration or less will be repaired.
      Note: Not active unless <remmaxolappct> is also active
    <repmaxgapmin> = the maximum minutes a gap may be to repair
      0 = disabled
      10 = default - any gaps 10 minutes or less will be corrected.
      Note: Not active unless <remmaxolappct> is also active
      Note: This option is safe as it always lengthens programs.
    <repmaxgappct> = the maximum percentage a gap may be to repair.
      0 = disabled
      25 = default - any gaps 25% of duration or less will be repaired.
      Note: Not active unless <remmaxolappct> is also active

8.2 (complete package)
  *Fixed Install.bat
  *Added Perth Configs, thanks Jolian
  *Split SBS Grabber into SBS1 & SBS2.
  *Fixed some of the location configs for sbs2 etc.  
  *Fixed Typo's  

8.1 (complete package)
  *Added Canberra Configs, thanks ozNick
  *Removed _ in directory names.
  *Created install.bat in each location.
    - Run the install.bat in your chosen location and all binaries etc are copied from the Melbourne config.
    - Delete undesired locations after this. eg. if in Canberra run the install.bat in Canberra, then delete Melbourne, Sydney etc.  
  *Added TV Scheduler Pro Directory settings.

8.0 (complete package)
  *Moved all sources to _location subdirectory and added location configuration option
    - Now locations can be changed by one setting in config.xml as long as source folders are present.
  *Added Sydney configurations - thanks to R Palmer.
  *Added TimeZoneDetectLowerLimit to prevent small (minutes) timezone corrections (adjustable).  
  *Added SBS to epgscan now that SBS is transmitting data.
  *EPGScan has 3 tries per channel on failure.    
  *Added checks to abcjc grabber for when data source has am/pm errors.
  *Changed sbsgrabber url for Sydney

Version 7 no longer supported.

Archive of Version 7 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V7.zip

CODE

7.10 (xmltv.vbs & config.xml)
  *Fixed bug in Timezone correction.
  *Changed some time correction programs in config.  

7.9 (xmltv.vbs only)
  *Improved overlap detection - (overlaps two directions & removed redundant enveloped 1.)

7.8 (Complete Package)
  *SBSGrabber updated to 1.0.3
  *D1 removed from sources.
  *Fix bug in Ice dtd download.
  *Added max overlap option to further improve overlap detection
  *Added output epg_source per programme in xmltv file in testmode  

7.7 (xmltv.vbs only)
  *Improved overlap detection.

7.6 (xmltv.vbs and guidedog.vbs, & icetv files)
  *Updated process handling to terminate ALL child process if they timeout
  *Updated IceTV downloader to grab the dtd file and save a local copy for when the server/internet is down.
    * (Allows validation and still be able to use cached xml files.)
  *Updated guidedog to 1.5.02

7.5 (Complete Package)
  *Updated all xmltv.dtd files to latest version (1.38)
  *Added resolve externals config option - (allows not parsing dtd files at all)
  *Added validate option allows skipping xml validation on load if desired.
  *Updated SBSGrabber to address 24.xx hour crash.    

7.4 (xmltv.vbs and config.xml)
  *Updated delimiters to "::" and ";;" to prevent clashes with xml escape codes.

7.3 (xmltv.vbs and guidedog.vbs)
  *Updated some log settings to display best/worst channel
  *Updated to guidedog 1.5.01  

7.2 (complete package)
  *Updated OzTivo to latest Guidedog and EPGStream methods. (Guidedog default)
  *Added ABC HD to epgscan channelmap

7.1 (complete package)
  *Added per source formatting options.
    - Upper/Lower and Title Case for Title only for now.
  *Added Summary Option to log file (saves scrolling through emails)
  *Added ABC HD, Nine HD and SBS HD to oztivo etc (although data not necessarily available)
  *Added xmltvDownloader to oztivo grabber for March changeover.
  *Changed IceTV download to get full descriptions.  
  *Ordered subs/funcs alphabetically.

7.0 (complete package)
  *Added HD channels to channelmap settings.
  *Added EPGScan grabbers (Shaun Faulds)
  *Added SBSgrabber (Nate)
  *Added option to ignore seconds to config.xml for old webscheduler version. <seconds>  
  *Added option to copy items from SD to HD channels if slots are empty. <channelpairs>
    *This can cause further overlaps etc, use with caution.
  * /B added to batch files.
  *Changed default channel names to not be Melbourne centric. ie Channel Seven Melbourne is now Seven.
  *Added option to exclude items based on title <exclude>

Version 6 no longer supported.

Archive of Version 6 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V6.zip

CODE

6.2 (xmltv.vbs &config.xml updated)
  *Added timezone correction limit incase false positive occurs with large offset.
  *Small bugfix in tzdetect.  

6.1 (xmltv.vbs & config.xml updated)
  *Added Empty method. Adds only if there is a match or the slot is empty. Use to reduce overlaps if desired.
  *Removed title strip option. This could already be done with the filter option, example included.  

6.0 (complete package)
  *Added Automatic TimeZone Correction. (Refer Config File for details)
  *Added Mode to Sources - Allows cascading runs without redownloading.
    -Full = Download new files and Process, Process = Process existing files only (Similar to testmode but per source)  
  *Removed JRobbo Sources.
  *Added IceTV source option.
  *Option to strip items from title. eg Remove Movie: for IceTV

Version 5 no longer supported.

Archive of Version 5 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V5.zip

CODE

5.3 (xmltv.vbs & config.xml updated)
  *Added a per source TimeZone offset to cater for sources that take a while to sort out DST rollovers. Refer config.xml

5.2 (Complete Package)
  *Added abcjc grabber version 1.0 to grab ABC & ABC2 a week ahead direct from abc.net.au
  *Various tidy ups.

5.1 (xmltv.vbs only)
  *Enhanced malformed source tolerance.

5.0 (xmltv.vbs & config.xml updated)
  *Added Unicode Support - thanks Uryan
  *Output charset is definable in config.xml, iso utf etc.
  *Filtering is definable in config.xml, can also be turned off.
  *Fixed logging of latest start time was incorrect.
  *Added logging of best and worst channel latest start times.
  *Added logging of each channel latest start times.

Version 4 No Longer Updated.

Archive of Version 4 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V4.zip

CODE

4.6 (xmltv.vbs & config.xml updated)
  *Added timezone override to config.xml options
  *Added tzoffset override to config.xml options.

4.5 (xmltv.vbs and grabber xmltv.bat files changed)
  *Replaced Filter code with much more efficient Regex Filter (thanks Uryan)
  *Simplified xmltv.bat files path statements.  

4.4 (xmltv.vbs and config.xml updated)
  *Moved Filterillegal Characters to output only to improve performance.
  *Moved channelmap file to config.xml to facilitate multiple regions.  

4.3 (xmltv.vbs and xmltv.bat updated)
  *Added Filterillegal Characters back in due to characters in myxmltv source crashing combiner.
  *Error in xmltv.bat reference to Matchlist directory for DVB Webscheduler Pro.
  *Added version to xmltv.xml file for reference.    

4.2 (xmltv.vbs and config.xml updated)
  *Bug with items with no subtitle, desc etc. (thanks uryan).
  *Moved mindur/maxdur and mround to config.xml
  *Shortened maxdur to 500mins.    

4.1 (Complete Package Required - Changed update and xmltv.bat)
  *New xmltv and update bat files with webscheduler free and pro detection.
  *Fixed error from early 4.0 release.  

4.0 (Complete Package Required)
  *New version 4.0 (AusEPG Build Start)
  *Added ability to specify configfile from command line.
     xmltv.vbs -configfile "C:\Path to configfile\config.xml"  
  *Added ability to override any config.xml setting from commandline
     xmltv.vbs -<config setting> <value> eg... xmltv.vbs -mindays 3  
  *Modified default myxmltv setting to be SubTitle instead of match.
  *Added DVB WS Pro matchlist paths.  
  *Some logging fixes.

Version 3 No Longer Supported.

Archive of Version 3 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V3.zip

CODE

3.6 (Complete Package Required)
  *Fixed error in config.xml for d1 changeover
  *Fixed bug in roll-over code.  

3.5 (Complete Package Required.)
  *Replaced xmltvdownloader with d1xmltvdownloader.
  *Improvements to xml extraction to suit.
  *Improved some of the bat files.
  *Changed oz-tivo to default to wget method.

3.4 (xmltv.vbs and config.xml updated)
  *Added config option <mindays> = the days to keep at the start of the guide.
       0 = Trim Now, 1 = Today 0:00am,  2 = Yesterday 0:00am etc.
  *Made some log items show zero.
  * Added error count at end for quick checking of email log.

3.3 (Complete Package Required.)
  *Made uryans oztivocached the default downloader for oztivo guide.
       Much faster, does not require an account and less load on minnietuhs servers.
  *Removed oztivograb method - no longer supported.
  *Fixed a small bug that could allow credits to keep getting added
    (only happened if safexmltv recursed its own guides.)
  *Tidied up logging, removed some old stuff.

3.2 (xmltv.vbs and config.xml update)
  *Added config option <minprog> = the minimum number of valid programs before rolling to the next
   cached guide. - Default 10. Added this for scrapers that write valid xml when they fail but contain no data.
   Can be set to zero to prevent failover.

3.1 (Complete Package Required.)
  *Substantial improvement to overlap detection.
  *Added missing xmltv.dtd files.

3.0 (Complete Package Required.)
  *Major upgrade over version 2. Version 1 method is now completely removed from the package.
  *Output file is now written per program with consistant layout and Timezone for Local Machine.
  *Start/Stop times are now fully parsed. Short times are supported and the package will default to UTC
    if no timezone is specified in the source xml files. This is done per program. All times are
   converted to full format yyyymmddhhmmss +tztz to ensure compliance with Webscheduler.
   Any programs with nonresolvable times are discarded.
  *XML channel mapping per guide source is now supported.
    - A channelmap.xml file is placed in each source dir. It contains the source channel name,
      The destination name and an offset in hhmm format. This therefore supports offsetting times
      Per Channel, and Per Guide Source.
  *Credits are appended to the description.
  *guide day rollover is done internally now so that guideday caching can be changed from the config file.
  *guide day caching checks the age of the files before rolling over.
  *Convertxmltv.vbs is removed - now obsolete
  *Default to wget for oztivo - runs faster for me and TZ and offsets no longer required.
  *Numerous other changes.

Versions 1 & 2 No Longer Supported.

Archive of Last Versions 1&2 available from.......

http://mail.lewis.com.au/public/xmltv/SafeXMLTV-V1&2.zip

CODE

2.7 (Complete package required and one new line in config.xml to add).
*New DateFail option in Config.xml - allows different handling of start/stop time problems.
        0 = Continue on and do not recurse other sources.
        1 = Continue on but then try previous days data - Default.
        2 = Terminate this guide immediately and then try previous days data.  
  *oztivo downloader updated to uryans latest and defaults back to this.
  *fixed bug in convertxmltv.vbs - for times not starting at 00s.

2.6
*New TimeZone Checking option in Config.xml - allows different levels of start/stop timezone checking
     0 = No checking - Not recommended.
     1 = Check that time zone is valid and ±2400 hrs.
     2 = Timezone must be valid and equal to Local or UTC.
  *More logging options, tidyups and misc minor bug fixes.
  *oztivo downloader can now choose between wget or uryan's script with one setting.
  *fixed bug in convertxmltv.vbs

2.5
  *New logging option to set logging level 0(none)-4(debug).
  *Stop parsing invalid guide immediately.
  *Overhaul of report layout for legibility.
  *Couple of minor bugs fixed (in logging).

2.4
  *New SubTitle data parsing setting per source in Config to prioritise by sub-title then description.
      This will give preference to EPG data that has data in the sub-title field.
      I often found that the longest description can be a generic one,
      while data in another EPG could contain a proper description,
      typically with sub-title data.
      If sub-titles are the same then the longest description is chosen.
      Also the old method can still be used.
  *Fixed bug with channel id's at the start of the file in version 2.3.

2.3
  *Separate config.xml file to simplify future updates.
  *Some extra logging.
  *Added some timeout detections to wget calls.
  *xmltvgrabber log cleanup.
  *reverted back to uryans oztivograbber, new version.

2.2 (xmltv2.vbs updated only)
  * Improved the performance of the overlap detection. (Done in a single pass).
  * Fixed bug in Match type.
  * Added maximum day limit.
  * Improved Logging.
  * Removed redundant code.
  * More tidy up and other fixes.

2.1
  * New and improved overlap detection.
    Looks for items that are enveloped by others and removes them.
    This seems to do a pretty good job of removing the extra items that cause WS epg to shift.
    It will only remove overlaps that are not in the matchlist.
  * Also added two types of data parsing per source-
    Description: will use all the data and overwrite based on longer descriptions.
    Match: will only use data from this source if its a matchlist item not already
                  picked up by the Description guides.
                  (I find using the myxmltv guide data caused too many overlaps and
                   gaps but running it as Match works well.)
   * Improved Character filtering (Had some issues with myxmltv data)
   * Updated myxmltv conversion script.
   * Other cosmetic improvements.

2.0
  Completely new version that creates a single guide from multiple guide data.
  It downloads the guides and compiles a new one based on items matching your matchlist,
  and then by the longest description.
  It then tries to remove as many duplicate items and overlaps as possible,
  but there may be of them than from a single source.
  Having said that it would not be missing any data, especially matchlist items.
  The package includes both Version 2 and 1.X.
  To enable version 2 simply change the xmltv.bat file to call xmltv2.vbs instead of xmltv.vbs.
  You can of course change back at any time.
  Version 2 uses all the same data downloaded as Version 1.X.

CODE

1.6
  Updated for more matchlists.
  Updated xmltvgrabber.
  Disabled myxmltv.

1.5
  Updated xmltvgrabber script to delete old cache files (until code is fixed).
  Compares guides with different timezone declarations properly (ie one declared vs utc).

1.4
  Updated oztivo grabber.
  Faster processing of guides (if invalid data is found).

1.3
  Looks for valid program durations and also uses distinct start times in weighting criteria.
  Selection criteria is now fixed as other options not worthwhile, criteria is as follows.....
  1. The most distinct (different) matchlist matches within the matchdays window (2 days default)
  2. If (1) is equal then the most total matchlist matches within the matchdays window.
  3. If (2) is equal then the best weighting criteria calculated as follows.....
    weighting = total matchlist matches (entire guide, after now) x
        total number of days in guide (after now) x
        total valid programs with different start times x
        total length of all valid program descriptions.

  * These changes were made due to current problems with jrobbo myxmltv guide
    (has lots of programs starting and finishing at midnight).
   A valid program is determined to be after now, longer than the min duration setting (default 5 mins) and
   shorter than the max duration setting (default 400 mins). It does not matter if there are legit programs
   outside these parameters, it just means they are not used in the weighting calculation as there should not
   be many programs < 5 mins or > 400 mins.

1.2 - Internal version.

1.1 - Added fix for *something* matchlist items

1.0 - Initial Release..
  Downloads xmltv files from multiple sources.
  Converts each source to have the same channels names.
  Checks each source for valid structure, valid start/stop times, valid timezone etc.
  Checks each source against your webscheduler matchlist and
  then selects the best file based on the following criteria.

   1. Most distinct matches.
   2. Most total matches.
   3. A ranking based - Refer version 1.4 above.
      It then checks this against the xmltv file from the day before and
          only updates if the new one is better.
   4. Stores 7 days worth of each downloader in case internet connection fails for a day or two.
   5. Emails a log to you after it has run.
   6. Updates Webscheduler if the data is better than yesterday.
   Lots of other things.

   Note: Add it  (xmltv.bat) as a scheduled task in Windows to run in the morning.
           And remove any xmltv scheduling from webscheduler itself.

  Just check through some of the files to make sure its configures for your own use.....
  eg .
    1. place it in C:\Program Files\XMLTV
           (Will work in other directories by changing all references to this directory).
    2. add your username and password to the oztivograb.vbs file
    3. add you email user address, mail server etc to email.bat
    4. if your webscheduler port 7 or directory is non standard then change this in update.bat etc.

    Once you have it running it is very easy to add additional sources to the system as you find them.
    I have 4 sources included of different types methods etc to get people started.

Good Luck.

This post has been edited by Calvi: Jun 2 2012, 01:09 PM

[Guest_budge31_*]

Hi Calvi,

Something weird is happening. For whatever reason, none of the data from xmltvgrabber (short description) guide is making it into the main guide. If I load the resultant xmltv file from the xmltvgrabber (short description), I get the data I am expecting, but it is not being blended in to the main guide.
I think it has being only doing this such the last update, but it could have been from th pevious to last update (as it was only around for 1 day).

Any Ideas?


On a side track, Does anyone know what happened to Battlestar Galactica this week? It wasn't recorded last night, and I can't find it in the guide anywhere. Its as bad as The Soprano's being moved to the middle of a Monday night without any warning (thank god for WS in that instance). Don't these dumbass stations realise that just because it hasn't got CSI or Idol in its title, doen't mean people don't watch it???

This post has been edited by budge31: Oct 26 2006, 09:57 AM