TSNowAndNext, Create ProjectX cut file from TS Now and Next information Options

[nate]

Here's another new program for you guys.

While experimenting with Comskip I discovered that the large pre and post padding i use with my recordings so that I don't miss stuff when the networks run late was causing a lot of unnecessary processing. I originally had the idea that using the now and next information should be added to Comskip, and to help out with that I set out to write some code to see how well it could be implemented. This is the result.

TSNowAndNext is a command line tool that creates a ProjectX cut file from the now and next information contained within the transport stream.

TSNowAndNext
Source code (Subversion repository)



There's a batch file included that I use which will load up ProjectX with the cut file. See the usage for details.
I'm also working on another program that uses a series of file watchers and batch files to run a ts file through TSNowAndNext, ProjectX, Comskip, Cuttermaran (including waiting for manual checking of cut points) and finally mencoder. It's still pretty raw though, but once I've got it working okay I'll share it here.

I'm happy to hear any feedback, bug, or feature requests.

[DrP]

A nifty tool if the recording includes the EIT.

Have you thought about using the event running status rather N&N events themselves? Australia's Freeview system relies on the event status flag to determine when an event is showing so in theory it should be more useful than the event positions. Here it's not at all unusual to see program B on the screen while the EIT N&N still shows program A but the running status may (have to say may because like most things Australian TV related, it's not a certainty) actually show which event is actually on screen. It confuses the crap out of some TVs, I've seen sets that when the info button is pressed it shows the Now event as blank and the Next event as the currently showing event.

Other countries blessed with more stable digital television systems such as the UK also make good use the event status.

This post has been edited by DrP: Nov 9 2011, 12:56 PM

[nate]

A nifty tool if the recording includes the EIT.

I would expect most .ts file would include the EIT packets. TS Mux recordings from WebScheduler/TV Scheduler Pro/DNTV Scheduler all include the EIT. The only .ts files I've tested that don't are the output of ProjectX remux operations that have stripped the EIT packets.

Have you thought about using the event running status rather N&N events themselves?

I hadn't thought of that. At the moment I'm just assuming that the first event in the list is the currently showing event. In all my tests that running_status flag of the first event has always been "running", but I guess I should add a check in case it's not.

I'm not using the date and time included with the N&N at all so that removes one source of confusion that some TV's might be dealing with. I had thought about using it, but was surprised to discover that not all transport streams include time_date_section packets (0x70). Without those I don't think there's a way to determine the "current time" in the file.

[DrP]

I would expect most .ts file would include the EIT packets. TS Mux recordings from WebScheduler/TV Scheduler Pro/DNTV Scheduler all include the EIT. The only .ts files I've tested that don't are the output of ProjectX remux operations that have stripped the EIT packets.

So it does, I've never taken enough notice to see that. I'll start giving your tool a run over some random test recordings and see how it goes.

[nate]

Hey DrP, I've made the changes you suggested, as well as fixed a few other exceptions I came across in testing.


New Version 1.1.0.12

Changes:

• Fixed exceptions when short_event_descriptor is not present
• Changed to use the event_id from the event with running_status==4 instead of section_number==0.
• Added Version information

[DrP]

The new version gives me a 404 error when I try to download it.

BTW, I ran a recording from 7mate through the earlier version just now and the N&N ticks over approx 4 mins into the show. When I can download the latest version I'll see how it goes.

[nate]

The new version gives me a 404 error when I try to download it.

Fixed. I accidentally mixed up my dashes and underscores.

[DrP]

OK, same file with the new version and it ticks over at a different location so that side is working but due to the broadcast its still out of whack with the actual program. I'll send an email to someone to see if it can be corrected. Might get lucky.

[nate]

After reading you post about 7mate I recorded a few short shows tonight. My previous testing had all been with TEN, Seven, and ABC and I hadn't seen it out by more then about 10 seconds.

7mate: N&N change was delayed by about 2 seconds at the start of How I Met Your Mother, and about 7 seconds at the start of Pawn Stars.
7TWO: N&N changed 5 seconds after the start of both Keeping Up Appearances and The Royal.
GEM: N&N changed 35 seconds before the end of Friends, and 12 seconds before the end of The Zoo.
SBS TWO: N&N changed 3 minutes before Iron Chef ended, and about 15 seconds before Letters And Numbers ended.

It's only a couple of tests, but GEM and SBS TWO look to be off by a lot more than the 7* channels. I figure we can deal with 10 seconds either way by adding some padding, but 3 minutes is way off.

[erikbrannlund]

I shouldn't rely to much on the event or the EIT or whatever. A few years ago I had implemented this in my recorder to start and stop the recording when the show started and ended, to handle delayed shows, but it didn't work. After some phone calls to the DVB network provider it turned out that they received the start and stop times from the "content providers" (on a discette if I remeber correctly) in advance and just sent the EIT and status flags according to the times in the file. This might have changed.

/Erik

[DrP]

After reading you post about 7mate I recorded a few short shows tonight. My previous testing had all been with TEN, Seven, and ABC and I hadn't seen it out by more then about 10 seconds.

I think its really going to come down to how the programming gets to an area. If its direct from the originating network - TEN, 7 etc - then I expect the change point will be reasonably accurate. If it goes through an affiliate (even a wholey owned affiliate such as NBN) there's potential for things to go wrong. Consistency seems to be a real issue, at least with how WIN, SC and 7 go to air here. Sometimes they are good, other times pretty hopeless.

Having said that a few tests I made overnight revealed:
WIN SD - change points happened in such a way that the show was entirely covered by the cut points with not much to spare
SC10 SD - about 30 seconds late
ABC2 - entirely covered
ABC1 - entirely covered
SBS1 - program failed to operate correctly, took a long time parsing the file then produced a cut list that included the entire file

Anyway, another suggestion for the program - a command line option to specify a string to search for in the EIT to select the program to extract rather than rely on percentages etc. This would allow extraction of multiple programs from a single recording file.

This post has been edited by DrP: Nov 11 2011, 07:31 AM

[nate]

a command line option to specify a string to search for in the EIT to select the program to extract rather than rely on percentages etc. This would allow extraction of multiple programs from a single recording file.

I can certainly add a search string, or another option is to have the program list all the programs in the file and ask which one to output.

I'm assuming you're not after it to be used in an automated way, so asking for user input is okay. I went with percentages to start with because I wanted it to run without interaction from my processing script.

It would look something like this:

CODE

opening D:\TV\Filename.ts

Searching for programs...
1)             0   0.00% Friends
2)     424940724  10.05% The Zoo
3)    3029535212  71.29% BABY PLANET

Please select a program: 2
writing D:\TV\Filename.ts.Xcl

[DrP]

Hmm. Looking at it from a scripting perspective how about when the command line search item is specified it runs without interaction otherwise it lists out and prompts for the program to extract?

[DrP]

I shouldn't rely to much on the event or the EIT or whatever.

Doesn't look like all that much has changed. A recording from 7mate today and the EIT ticks over 3 mins before the show started and ticked again 5mins before it ended.

[nate]

In case anyone is using this, I've made some minor fixes.


New Version 1.1.0.19

Changes:

• Fixed problem parsing EIT event when the events section is longer than 255 bytes.
• Added a message when printing events to warn about long gaps without Now EIT packets.
• Added -percentage command line option for specifying where in the file the show is.

[DrP]

I'll give it a whirl tonight.