Mfs_Ftp: extract, archive, restore & transfer recordings

[rc3105]

here's a little tivo side ftp server for moving recordings from tivo -> pc, pc -> tivo or tivo <-> tivo with a regular browser or ftp client. video can be moved between different versions of tivo hardware / software and streamviewed on a recieving tivo

mfs_ftp.tar.zip - current version of the complete program dir including s1 support binaries
(attached to this post)

installation:

shutdown any previous version of mfs_ftp running in the tivo

unzip the mfs_ftp.tar.zip to mfs_ftp.tar on your pc

open mfs_ftp.tar & extract readme.txt - READ IT!

transfer mfs_ftp.tar to /var in the tivo

backup the old mfs_ftp dir: "mv /var/mfs_ftp /var/mfs_ftp.bak"

from the "/var " prompt unpack with

"tar -xvf mfs_ftp.tar" or "cpio -i -H tar -d -F mfs_ftp.tar"

start from bash "/var/mfs_ftp/mfs_ftp.tcl" or load from a startup script


mfs_ftp 1.2.9 MAY NOT BE REDISTRIBUTED. PERIOD! (read the license and terms of use)

sadly many are offering mfs_ftp installed on units for sale, since the legality of the license is unlikely to deter the unscrupulous here's the deal. if you want to include mfs_ftp on a tivo your selling (presumably as an inducement to buy) send your contact info with a $40 paypal to rc3105@hotmail.com I'll provide a transferrable license tied to the tivo itself & permission to install / distribute mfs_ftp on that particular tivo. this provides a legit means for techs to install sw for the clueless & specified damages that can be recovered (in court, etc) from those distributing in violation of copyright

my paypal is rc3105@hotmail.com and donations are allways appreciated. mfs_ftp is FREE for personal non-commercial use if you install it yourself, so enjoy


Usefull things to have / know about

THIS THREAD IS THE FAQ! (and moderated to stay that way)

please read readme.txt & this ENTIRE thread BEFORE posting a mfs_ftp related question - off topic posts, obsolete info & chitchat more than a month or so old gets nuked

please ask any questions re: installation / usage in this support thread instead of cluttering the forum starting new ones


ALL required S1 support files are included in the zip

here's links to usefull alternate threads

MFS_FTP on Series 2 Units contains S2 support binaries - substitute s2 binaries after installation but before starting mfs_ftp

Mfs_Ftp: suggestions, feedback, etc

mfs ftp issues in the newbie forum

tmf / ty+ tivoweb extraction module: extract tmf or ty+ via tivoweb

tmf2ty_cfg.exe is a windows binary that extracts ty from tmf. these ty are insertable with mfs_ftp & compatable with all ty utilities. if savepath.txt is in the directory with tmf2ty_cfg.exe (or in the dir with the tmf under some versions of XP) the first line of the file is used as the output path. these are tcl paths so use forward slashes - "x:/extracts/" "d:/my_recordings/" "v:/video_files/" etc

Commonly asked questions (and answers)

[wheelbarrow]

Nice one Guys, this is the best TVIO app i have got. Can you please pass thanks onto all the clever people who made this possible.

Ok, I'm having a "Blonde" moment, I;ve copied the 3 files to /mfs-ftp, made them exce with chmod. but when i do ./mfs_ftp.tcl i get "No Such File or Directory" ?

I have missed somthing here, i can run the app by doing /tvbin/tivosh msf_ftp.tcl, but it stop when i try to download as it cant run the other file.

Please call me stupid as i'm sure it a very silly mistake that i have made.

Regards

Paul

[rc3105]

set your ftp client for binary mode whenever transferring via tivoftpd.

a tmf is just a tarfile. winrar & some other utils can only see files in the first 2 gig of a tar. winzip works fine

many ftp clients will only download the first 2 gig even if you have fat32 (4gig limit) or hpfs, ntfs, ext2/3 (no limit) please make sure your ftp client isn't broken

[][ronMan]

So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone. I had suspected this would be the case from earlier work I was doing with insertion. Overall, things look great.

What I'd love to be able to have is a tool that I could give an AVI, MPEG, etc along with some details about it, and could then covert that to the TMF format for insertion onto the Tivo. i.e. Download a movie trailer from the net then insert it on the Tivo.

Keep up the great work...

Derek aka ][ronMan aka DSmooth

[rc3105]

ok, couple of people have asked, so here's the info.

tmf stands for tivo media format. the NAME was AlphaWolf's idea :-p the original code to generate it was done by tivodvlpr & mbm.

edit: sorry, how did I forget to mention dtype? duh@me...

a .tmf file is a regular tar file that contains the following, in this order.

showing.xml
part00.ty
part01.ty
...
part04.ty


showing.xm is a xml file with the info from the nowshowing database stored as regular text. title, episode, description, everything. the .ty parts each represend an individual fsid as it's stored in the tivo.

pretty simple really. the xml is a bit hairy, but easy enough to parse out the title / description etc and it contains all the info the tivo stores about a recording.

--
Riley

[][ronMan]

Originally posted by ][ronMan
So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone.

Correction... With the 1.1.1 version I am also able to ftp a show off of my DirecTivo and ftp it onto my series 1 standalone. This is really starting to rock!

Derek aka ][ronMan aka DSmooth

[][ronMan]

Originally posted by AlphaWolf
DT wouldn't work on an SA because of the fact that its very non-standard. Again, doing this would involve being able to generate tystream archives.

Also another comment on the binary optimizations: It would be better to do it anyways, b/c you could further minimize the chance of pausing or any other errors.

See my last post... This is working OK with 1.1.1 and greater...

Derek aka ][ronMan aka DSmooth

[stealthdave]

Originally posted by rc3105
PASV is most likely the problem it's still a bit iffy. there it was trying to connect to itself. use regular mode instead.

Sure enough, as you were posting this I had just tried turning off passive mode, and that fixed the problem. My guess is that's what the majority of the client-specific problems are (I can't be sure of course, but it seems likely).

Would it be possible to add a short "how-to" to the readme.txt file? IMO, it should include the following:[list=1][*]command-line server configuration[*]turn passive mode OFF[*]filenames (i.e., command-line clients need only use the FSID listed as the filename to get)[*]importing vs. exporting[*]deleting programs (and how to prevent it (?) )[*]other[/list=1]
Might save a lot of repetitive questions. Btw, fantastic work!!! This blows away my old export-via-nfs method! (For some reason nfs is really slow on my Tivo, even with rsize=8192&wsize=8192.)

- Stealth Dave

[][ronMan]

Originally posted by alanjudy
The other question is, while I have seen that someone has gone from SA to DTivo, has any sucessfully gone from DTivo (2.5.2) to SA (3.0.1)

Thanks

=alanjudy

alanjudy,
This is my exact setup, and as you see from my post above, with versions 1.1.1 and beyond it works in bot directions...

Derek aka ][ronMan aka DSmooth

[Hi8]

Well; I'm xfering a .tmf originally from my dtivo to my SA. It's the same one that created my 0Byte file, undeletable listing. I found out that my SA is MUCH slower to respond and transfer to. Both tivos are 3.1 - I just didn't WAIT long enough for it to allocate space .. It is a 2hour movie, so it appears it;s going to take some time. I'll report back on results.

here is a dump of the transfer sofar...

echo to verify: PORT 192,168,1,49,7,156
ip is 192.168.1.49 port is 1948
200 PORT command successful.
echo to verify: STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
entering dumpxml
segment is showing.xml
leaving dumpxml
entering parsexml
title is "Hackers" - episode "unknown episode"
leaving parsexml
attempting to create recording - success
attempting to set recordingfsid - success
attempting to set bitrate & such - success
attempting to create a dummy station - success
attempting to create a showing - success - trying to set showing info - success
trying to create the program & set the info - success
attempting to showing set program - success
created Hackers : unknown episode : fsid# 2842851
entering tmf2fsid
starting segment part00.ty
entering AddPart
trying to open recordingfsid
success
createing recording ? success
trying to add part - success
trying to mfs streamfill allocate space
success - created newfsid 2842854
trying to set file to fsidtwo - success
leaving AddPart

[rc3105]

the initial space allocation can take bit, especially if the db is busy & it has to retry a few times.

I'm working on that issue

set your ftp client timeout to at least a minute, 2 ought to be worst case.

--
Riley

[Hi8]

Originally posted by Hi8
I'll report back on results.

It works perfectly! Plays and looks as good if not BETTER than anything ever recorded on my SA, well, there you have it!

the down side is xfer speed. it took about 5hours to xfer a 1h45m movie to my SA. What is surprising is how fast it downloaded from my dtivo, less than 25mins.

I noticed the time difference the listing info reported a 1hr recording yet the green progbar reported correctly at 1:45.

Great Job RC!

here is the ftp sessionlog info:

sending Hackers .tmf as Hackers .tmf (1 of 1)
PORT 192,168,1,49,7,156
200 PORT command successful.
STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
Transmitted 1920999936 bytes in 17739.4 secs, (1.04 Mbps), transfer succeeded
226 File transfer complete
PWD
257 "/" is current directory.
TYPE A
200 Type set to A.
PORT 192,168,1,49,8,109
200 PORT command successful.
LIST
150 Opening data connection for file list.
Received 1068 bytes in 0.1 secs, (100.00 Kbps), transfer succeeded
226 Transfer complete.
QUIT
shutting down & exiting


**** please note the last entry ****

on my SA it shuts the server down at the end of a session, on my dtivo it just returns a 221 call 'server closing control connection' the former is abit bit of a disadavantage as I have to go in and manually restart it for each session. On my dtivo it starts on bootup, and runs along side my tivoftpd only on port 24.

using the same client WS_FTP Pro, and both my tivos are running 3.1, and mfs_ftp 1.1.2

[AlphaWolf]

ok heres an explanation of the file formats

.ty is a plain binary dump of the individual tystream parts combined into one file. This is the legacy method of extracting tystreams, and its my personal recommendation that nobody dumps/extracts/archives to this format anymore.

.tmf is a tar archive of the above, only a few additions. Each part of the tystream is stored as an individual file inside of this tar archive. On top of that, there is an XML dump of the recordings guide data. This provides a lot of the flexability needed for insertion, and can also be used to help automate dvd authoring.

That said, converting between the two is simple. To convert from .tmf to .ty, you can just untar the archive (winrar does this), discard the showing.xml file, and merge each of the .ty parts into one file. To go from .ty to .tmf, you would need to break apart the tystream parts at their boundaries and into individual files, and generate the xml data by manualy filling in the title information, description, genre, etc etc etc, as this is not stored in a tystream dump.

There is one exception to this however. Any tystreams that were extracted using jdiners tytool will have issues with the master chunk. These streams may be converted to .tmf, but they probably won't be able to be reinserted into a tivo properly, because they were not extracted properly to begin with. This is an issue with the way tserver exports tystreams.

[rc3105]

Originally posted by laserfan
I'm still confused about why/how tar_server (tivoweb module) is used in all this. Why is this needed? It's mentioned in your comments/notes RC but there's no discussion of what module is used for what. Is Tivoweb supposed to be running when you do an ftp transfer to-or-from?
Sorry, but I can't sort out how this thing works.

mfs_ftp uses the same xml dumping routine (tcl) and the same mfs_tarstream (binary) as tivodvlpr's tivoweb tarserver module, the idea being that files produced are identical and interchangable with the (eventual) tivoweb insert module

tivoweb mfsstream produces regular .ty files.

mfs_ftp does not need tivoweb running.

[zobetron]

For what its worth,

I just downloaded a 5.59 GB (6,008,352,768 bytes on disk) to my ntfs formatted drive. When I opened it up with winrar, it (winrar) said the file was corrupt. I got a similar error with winzip.

So I tried a couple of other windows based decompressors and they worked just fine. (Specifically 7-Zip File Manager from source forge (free) and WinTAR-SCSI (trial)).

Z