Computer People - ideas?

[hyunelan2]

Here's the situation. Locally on a server here, I have 31GB of GIS data (various forms, GIS, Databases, images) that needs to be accessed in another location. The connection to this other location is poor, because of the geographical location, it is provided by a 2.4GHz point-to-point wireless bridge (3 miles away).

The problem is, loading this data takes forever. To open ArcGIS, for example, it takes over an hour to make all the database connections and open the program.

Here's what I want to do:
Since it is not vital that the data accessed remotely be live, I would like to setup a schedule for the new/changed data to be copied from my local server here, to a computer acting as a fileserver at the remote location. Weekly would be fine. What is the best way to do this in a way that is automated and I never have to click and drag things around every Monday?

[jalmir]

can you do that with a batch file???

IE :

"c:\Program Files\-the software flder-\-the software .exe-" -autoupdate(or wichever option that automatically launch the sych)

of course if the software don't have an exe option to automatically sych the files on open you'll have to find something else!


EDIT : you also have the Autoit option, it's quite powerfull

http://www.autoitscript.com/autoit3/

[03GT]

Here's the situation. Locally on a server here, I have 31GB of GIS data (various forms, GIS, Databases, images) that needs to be accessed in another location. The connection to this other location is poor, because of the geographical location, it is provided by a 2.4GHz point-to-point wireless bridge (3 miles away).

The problem is, loading this data takes forever. To open ArcGIS, for example, it takes over an hour to make all the database connections and open the program.

Here's what I want to do:
Since it is not vital that the data accessed remotely be live, I would like to setup a schedule for the new/changed data to be copied from my local server here, to a computer acting as a fileserver at the remote location. Weekly would be fine. What is the best way to do this in a way that is automated and I never have to click and drag things around every Monday?

Google "SyncToy" it's a free add-on for XP for syncronizing files. You can put it in your task scheduler to run any time you want. This is all GUI based so you don't have to be a scripting guru to get it done. Going with a script you'd have to put a crapload of switches on to do the verification, date stamp check etc. Since you'll be dealing with thousands of .sid files you want to make sure that you can verify them. Don't bother with trying to compress .sid files, it's virtually impossible. If you're not using XP there's a free-ware product out there I think it's called filesync.

[hyunelan2]

Ah, yes SyncToy. I played with that when it was first released, then forgot about it. The computer that will play host to these files in the remote location is an XP machine. The server it will get the files from is a 1TB Adaptec Snap Server, running the GuardianOS, but that shouldn't matter as it's in Windows emulation mode.

Right now, what I just did was copy over the whole directory that will be needed to a 250GB USB hard drive. After lunch I'm going to take it to the remote site and see if I can get synctoy to work ok. If that doesn't work, my fallback might just be to swap USB hard-drives out weekly... but that's inconvenient, a PITA, and is something I'd actually have to "do" rather than have the computers copy files themselves.


Thanks for the tip on synctoy. I'll post what results I get later today.

[SWortham]

You mentioned databases are involved. If it's a SQL database then I'd suggest setting up scheduled log shipping or differential backups. Using one of those methods you can drastically reduce the amount of data you have to transfer.

Also, there are some applications out there that will handle differential synchronization for file transfers. I don't know if SyncToy is one of them. But the idea is that it's possible to only send the bytes of a particular file that have changed, rather than transferring the entire file. The iBackup.com off-site backup service is one example of that technology. I'm sure there are others.

[03GT]

Yeah seagate's backup exec (don't know who owns it now) use to have an "open file" option that allowed for databases to be backed up even while in use. As long as the DB isn't in use it should be able to be backed up, that's why you have backups running at weird hours when nobody is around. The best solution would be to put something like Terminal Services / Citrix in then you don't have to do the data transfer crap. The data is in one place. But, that does require licsensing and it aint cheap.

SyncToy essentially syncronizes the files, so which ever one is newer is the one that's copied. That said, once you do the initial copy, synctoy copies the files that have changed and does not copy those that havent. But it copies the whole file, not just the changes. So if your DB is really big, that's going to be slow. Typically, differential/incremental are only used in terms of "backups" no in terms of syncronizations.

He's also kind of limited to what he can do with an XP workstation as the "server". I haven't played with MySql enough to know if you can set up some kind of daily data replication process on it. (assuming that's what he's using)

[SWortham]

Yeah seagate's backup exec (don't know who owns it now) use to have an "open file" option that allowed for databases to be backed up even while in use. As long as the DB isn't in use it should be able to be backed up, that's why you have backups running at weird hours when nobody is around. The best solution would be to put something like Terminal Services / Citrix in then you don't have to do the data transfer crap. The data is in one place. But, that does require licsensing and it aint cheap.
It isn't a challenge to keep the database up (at least with MS SQL) while backing it up. MS SQL supports that natively. It's only a challenge to keep it up during a restore. But it doesn't sound like that'll be a concern in this case. Like you said, if the backup/transfer/restore process takes place at 3AM, for example, then there should only be a brief period of downtime on the remote server when nobody should be using it anyway.

[yamaha]

You seem to be on the right path here.

[hyunelan2]

Ok, just got back from implementation. It now takes only a minute or two for the program to open and make all database connections, whereas before it was taking about an hour-and-a-half. So that's good.

I installed SyncToy on the XP box acting as a fileserver. I am using the "echo" settings on synctoy, as there are no changes to be made from the remote side, just viewing of the data. So if anything changes on the sever side, it overwrites the remote side. These are not SQL databases, but mostly geodatabases and file-geodatabases, in addition to .TIFs, shapefiles, and some Access Database information. I got the basic MS Windows scheduler to run synctoy overnight at 3am.

On testing, it did run successfully, but as expected certain files (like lockfiles) will not copy over. I don't see this as a problem though (not yet). On testing ArcGIS opened fine and saw the information that it needed to without any problems. Of course, with the "echo" settings, the remote location cannot do any editing, because it will just be overwritten from the server side at 3am. That's fine though, at the current time there is little need for editing at that site.

Everything would just be so much better if there were ANY telecommunications infrastructure around that site. No telephone lines capable of carrying DSL. No cable in the ground from Comcast. It's also near a river, and has bluffs surrounding it. Even building the wireless point to point was a PITA to bounce the signal around the prohibitive geography.

Anyway, 03GT gets the gold star for today. +1 point.

[03GT]

Glad I could help. :D