Distributed FTP
1. Introduction
DFTP is a multithreaded package; consist of a DTP Middleware
and an DFTP client. Goal is to distribute the file download among participating
client computers. DFTP Middleware provides the services necessary for
distributing the file among computers. DFTP client is a multithreaded client
application, which works in two modes, Using DFTP Middle ware, as well as a
standard FTP application if DFTP service is not available.
2. DFTP Client
Client works in two modes
In this mode the user can download
files from the ftp server as well as upload file to the server like a standard
FTP client. Multithreading enable the client to downloaded files in multiple
streams if supported by the server. Multiple files can also be downloaded at
the same time. Mutex technique allows only one instance of the application.
Internet Explorer Context menu
extension enables the user to directly select the file for download by
right clicking on the file and selecting “Download Using DTP” option. The
client will be featured with an attractive non rectangular skin. Skin changing
option enables the user to customize the skin.
DFTP Client will have all the
features of the standard ftp. Added advantage of using DFTP will be files can
be downloaded in multiple machines at the same time and then can be assembled
on the parent machine after download is completed. The user can also customize
the amount of download by each machine. “Download – No Transfer” if set will
download the files in the participating machines but will not transfer the
files to the parent machine, this is useful when u are downloading a huge file
and the clients are distributed over the internet.
3. DFTP Middleware
DFTP Middleware is the core of
DFTP Application. DFTP Middleware sits in between the client and FTP protocol.
DFTP Client can access the DFTP Middle ware by using COM interfaces which
allows the client to be language independent. Implementing the DFTP as a DAEMON
on a specific port is also under consideration allowing non MS programmer’s
java programmers to utilize this service. Target platform is Windows Machines
as they are the most widely used.
4. High Level Architecture
5. Modes Of Operation
- Multiple Clients downloading a single
file from a server from same different
- Multiple Clients downloading a single
file from a server from different network
6. Technical Details
DFTP CLIENT
Protocol:
FTP, DFTP
Language: VC++
Version: 1.00
DFTP MIDDLEWARE
Programming
Language: VC++
Service
Provided: DFTP
Version: 1.00
7. DFTP Web
DFTP web page will be a standard
static HTML web site where users can download the DFTP application.
Implementation of a simple registration system is also under consideration (if
time permits). Registration system would keep tracks of the users and will
allow the application to be used only by users who has a valid registered
product. DFTP is freeware; this would be the first step in designing and
developing anti piracy software applications.
8. Licensing
DFTP is Open Source, Freeware
users are free to use the application, modify the code.
9. Use Cases
Note: Use cases below do not
follow any specific standard or format. The interaction between the client and
the server is also represented in the form of use cases.
Case 1:
10. Class Diagram
11. DFTP FUNCTION
Client Function
1. DownloadFile(URL, Username , Password,
Protocol, Merge Flag, (Participating Client IP , Number of bytes to download)*
)
Note: If the server
do not allow download by parts FTP protocol with single thread is chosen.
|
URL
|
* URL of the file to download
|
|
User Name
|
User Name if
required
|
|
Password
|
Password if
required
|
|
Protocol
|
|
|
Merge Flag
|
- 0 – Do not merge file after download,
leave the files in the respective clients
- 1 -
Merge the files using P2P
|
|
Participating
Client IP, Number of Bytes to download
|
IP Address of
participating client as well and the number of bytes that the client needs to
download. If IP numbers are not specified the client starts downloading using
FTP protocol on the same machine.
|
2. GetStatus()
Returns:
(Number of packets,
(Packets)*)
Packet Structure
|
IP
|
IP address of the
client
|
|
Status
|
Online / Offline /
Download Complete / Error Number
|
|
Client Name
|
To Identify user
by name, User can specify it in the
client
|
|
Bytes Downloaded*
|
There can be n
number of Bytes download structures
- Local Path
- File Name
- Bytes Downloaded
|
|
Bytes Downloading*
|
Identifies the
current download initiated
|
|
Bytes Remaining
|
|
3.
SetDFTPProperties(struct DFTPPropertiesStruct)
Sets the DFTP
properties for the client middleware
4.
GetDFTPProperties(struct DFTPProptertiesStruct)
Gets the DFTP
properties
DFTPPropertiesStruct
|
Client Name
|
To Identify a
client by name , need not be unique , Internally IP is used to identify the
client
|
|
Number of
Downloads
|
Number of
simultaneous download allowed
|
|
Number Of Threads
|
The number of
threads the client needs to spawn for each file
|
|
Default Directory
|
Default directory
to download the files
|
|
Part Size
|
Part size for each
part downloaded
|
|
Cache disk space
|
This specifies the
amount of disk space DFTP can use.
|
|
DFTP Cache
|
This directory is
used to download the files requested by other computers.
|
5.
ImportFileAndMerge()
Imports the file
from the directory specified the user, Copies the file to the directory
identified by the structure and merge the files.
DFTP Middleware
Functions
GetConnectStatus(IP);
Returns:
|
Status
|
ON / OFF
|
|
Internet Status
|
Busy / Idle
|
|
Transfer Rate
|
Average Transfer Rate Bytes / Sec
|
|
Connections Available
|
Number of connections available for download. Eg. If the
user has allowed 3 simultaneous connections and if one is in use then it
returns 2.
|
CheckIfDownloaded()
This function checks if the file has is already been
downloaded by any of the clients. The function returns whether the file can be
downloaded from the local cache.
DistributedDownload
()
The Distributed Download function creates a “Middleware
Download Table” in the Host Middleware. The table is distributed among the
participating clients. Every client will hold the copy of files that are to be
downloaded by the client. The DistributedDownload() calls the SplitFile() to
calculates the best option to split the file, Table is populated with the
feedback from the client.
“Middleware Download
Table”
struct PartDownload
{
char[] szPartFileName
int iStartByte;
int iLength;
int Status
(Downloaded, Downloading, Queued)
PartDownload * PartDownload;
}
struct ParticipatingComputer
{
int iClientNumber;
char[] szFileName;
int iStartBye;
int iLength;
int iNumberofSimultanesDownloadsAllowed;
int iTotalNumberofParts;
int iTotalNumberofPartsDownloaded;
PartDownload* structPartDownload;
ParticipatingComputer
structPartComputer;
}
struct MiddleWareDownload
{
char[] szURL;
char[] szUserName;
char[] szPassword;
int iNumberOfParticipatingClients;
int iTotalByesDownloaded;
int iTotalByesDownloading;
ParticipatingComputer*
structPartComputer;
}
SplitFile()
Split file function calculates the how to distribute the
file efficiently and assigns a timeout for each part.
UpdateTable()
Update Table is called by the middleware when the time out
is reached for that particular client. The client sends the table to all the
other participating clients. If a client fails to send a table and is detected
by one of the clients then the client uses GetConnectionStatus() to determine
if the connection is active.
Connection Not Active
Case1: if Merge File Flag = 0 (Do not Merge)
The remaining non downloaded portion of the file set to
download from the inactive participating computer need to be downloaded. The
remaining portion is distributed among the participating computers. The
Decision for distribution is taken by the participating computer which has the
least “participating client number” (In most of the cases this is going to be
the client that initiated the download.
Case1: if Merge File Flag = 1 (Merge Files)
The participating computer checks the merge file class to
determine how much of the files has been transferred to the initiator client,
the remaining non downloaded portion of the file set to download from the
inactive participating computer need to be downloaded. The remaining portion is
distributed among the participating computers. The Decision for distribution is
taken by the participating computer which has the least “participating client
number” (In this cases this is going to be the client that initiated the
download.
FILE HANDLING:
Director Naming:
A directory is created for every file downloaded. All the
participating clients should have File Name +
Client Number as the directory name. If directory already exist, append
zeros to the directory name. Directory name for a file has to be same in all
the participating computers.
File Naming:
For example if Download File Name is DownloadFileName.EXT
Participating Computer will have the names for file as
Download File Name + (Participating Computer Number * 10)
Participating Client 1: DownloadFileName10.EXT
Participating Client 2: DownloadFileName20.EXT
Participating Computer Part File Name
For Client 1
Part1 :
DownloadFileName10100.EXT
Part2 : DownloadFileName10200.EXT
Part2 : DownloadFileName10300.EXT
Last Zero’s is left for download part by parts, in case one
part fails to download and remaining parts are downloaded. In this case the
download part is split again for download.
12. Conclusion
Objective of DFTP project was to
learn different technologies involved in creation of a stand-alone application,
Middleware. UML & Project management techniques where also explored which
was a part of my Masters Course work. DFTP would be the first FTP Client to use
distributed middleware technology.
13.
Future Considerations
- Distributed
FTP server with multi part upload.
- Detection
of client automatically within the Intranet.
- Client
List support to be provided by server.
- Algorithm
enhancements to distribute the clients.
- Algorithms
to use effective space utilization.
- Digital
Rights Management Support.