Author: bleny Date: 2010-07-01 12:38:20 +0200 (Thu, 01 Jul 2010) New Revision: 93 Url: http://nuiton.org/repositories/revision/diswork/93 Log: maj documentation, zargo Added: trunk/src/site/rst/diswork-daemon/devel.rst trunk/src/site/rst/diswork-daemon/draft.rst trunk/src/site/rst/diswork-fs/devel.rst trunk/src/site/rst/user/how_to_use.rst trunk/src/uml/ trunk/src/uml/diswork.zargo Removed: trunk/src/site/rst/devel/ trunk/src/site/rst/diswork-daemon/how_it_works.rst trunk/src/site/rst/diswork-fs/how_it_works.rst trunk/src/site/rst/user/install.rst trunk/src/site/rst/user/starting.rst Modified: trunk/src/site/site_en.xml Copied: trunk/src/site/rst/diswork-daemon/devel.rst (from rev 92, trunk/src/site/rst/diswork-daemon/how_it_works.rst) =================================================================== --- trunk/src/site/rst/diswork-daemon/devel.rst (rev 0) +++ trunk/src/site/rst/diswork-daemon/devel.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -0,0 +1,96 @@ +======================== +How Diswork Daemon works +======================== + +Abstract +-------- + +This document is for people who want to develop Diswork Daemon. + +Overview +-------- + +Diswork Daemon uses Diswork File System. + +Submit a job +------------ + +First, a node, before joining a network generate, for himself, a unique ID. And +then, create a directory "/home/ID". If this is not the first time the node +is run, ID should be reused so jobs can be found back after disconnect/connect. + +In his home directory, a node can put and remove his jobs. + +To create a job, a node should : + +#. Check that the application in the good version is available at + /bin/application/application-version.jar + +#. Create a directory in home for each of his jobs + /home/ID/jobs/job-1, /home/ID/jobs/job-2... + +#. In a job directory, put job.jsdl, a command line to run. This file contains + necessary information like : + + * The name of the job + * The application name, version + * File staging (needed file and where to get them) + * ... (to be completed) + +#. Declare the job as available by adding a symlink in /var/jobs/TODO + with the link named after the current date, the id of the job owner and + the job name. Places possible in /var/jobs/ are : + +#. Check the content of /home/ID/job-i/status + +#. Finally, clean home by deleting the directories of finished jobs + +Perform a job +------------- + +To perform a job, a node should find a job in /var/jobs/, this directory has +5 sub-directories : + +- /var/jobs/todo + +- /var/jobs/failed_1 + +- /var/jobs/failed_2 + +- /var/jobs/failed_3 + +Three more directories are available : todo_running, failed_1_running, +failed_2_running. + +#. Choose a job in one of those directories except "done" and "failed_3". + Use the data in the link same to know which job should be done first; + +#. Once the job is chosen, move the symlink to the "running" dir where you + found the link. Then, follow the link to get the actual data needed for + the job; + +#. Run the job; + +#. Now there is two way to proceed : + + - If the job successfully ran (no exception raised and return code is 0) : + + #. write the results in the job directory; + + #. mark job as finished + + - If the job has failed (a job taking too much time may be considered as + a failure). move it to according to where the job was found. If + the job was in "todo", move it to "failed_1" (read "failed once") ; + if the job was in "failed_1", move it to "failed_2", if it was in + "failed_2", move it to "failed_3". + + In both cases, write what occurred in the job directory in the status.txt + file (append your message, don't remove anything); + + +Additional step : before choosing a job, check in the all the "running" +directories if any symlink is not too old (use the name of the link). +It may means that a node crashed while processing this job. Move it to +the RUNNING directory of the next failure level ("todo" → "failed_1" → +"failed_2" → "failed_3") and try process it. \ No newline at end of file Copied: trunk/src/site/rst/diswork-daemon/draft.rst (from rev 92, trunk/src/site/rst/devel/draft.rst) =================================================================== --- trunk/src/site/rst/diswork-daemon/draft.rst (rev 0) +++ trunk/src/site/rst/diswork-daemon/draft.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -0,0 +1,84 @@ +Diswork first draft (fin mars 2010) +=================================== + + +Principe +-------- +3 types de machines : annuaires (uptime très grand), stockage (beaucoup de HDD), +calculateur (beaucoup de CPU). Un noeud (une machine physique) peut être d'un ou +plusieurs type. +Les annuaires sont reliés entre eux en DHT (Distributed Hashtable). +Potentiellement 1 ou 2 serveurs existent pour indiquer les point d'entrées +dans le réseau pour pouvoir fonctionner hors d'un réseau local. +Un client envoie aux annuaires les simulations qu'il doit effectuer : version du +simulateur et data. +Quand un calculateur a du temps, il demande à l'annuaire ce qu'il doit faire. +Récupère une simu, le simulateur approprié si il ne l'a pas (P2P ?) fait tourner +la simu et dit aux stockages que les résultats sont dispo. Réplication des +résultats entre les stockages. Client demande à l'annuaire où sont les résultats +et les télécharges (P2P ?) + +Storage +~~~~~~~ +Pour les serveurs de stockage, il recherche a duplique l'information +qui ne l'est pas assez. Par exemple dans l'annuaire, un resultat est +marque comme ayant une disponibilite de 1 (une seul machine active a +l'avoir) alors un serveur de stockage au repos va essaye de recuperer +ce resultat pour augmenter ce taux de partage. + +Il faut bien penser que l'intelligence est dans les serveurs qui +veulent faire du travail, pas dans un serveur qui repartira les choses +(surtout pour le stockage on se n'est pas une approche vraiment +naturelle) + +Tracker +~~~~~~~ +Les trackers possèdent les logiciels de simulation. + +Détails +------- +- Utilisation d'algorithme efficace pour découvrir l'ensemble des noeuds +- Un noued peut se contenter de ne connaitre que ces voisins (annuaire, stockage, calculateur) +- Un peu de config pour que le calculateur puisse choisir les applis qui ont le droit de tourner +- Quand résultats sont produits, aucun calculateur ne peut plus démarrer la simu +- Il faut optimiser les simulations a faire pour que le maximum de calculateur en fasse le plus de differente +- Plusieurs calculateur peuvent effectuer la même simulation +- Les calculateurs gere eux même les simulations en erreur (telle simulation a echoué 2 fois, je ne la fait plus) +- Le client peut télécharger les résultats depuis plusieurs noeud de stockage (P2P) +- Communication : utiliser HTTP pour éviter les firewall +- Vérification d'un ckecksum sur les logiciels de simulation (et les résultats?) +- Voir comment faire entrer Caparmor dans le réseau (noeud qui délègue à Caparmor ?) +- Mettre en pause la simulation quand l'utilisateur se ressert de son poste +- Detecter que l'utilisateur ne se sert plus de son poste pour lancer les simulations (inactivité, pas de charge cpu) +- Postes windows, utilisation d'un screensaver ? (diswork en cours) +- Les noeuds savent se mettre à jour (ou détecter qu'il ne sont pas à jour) +- Bonne gestion de l'espace disque sur les noeuds de stockage +- Systèmes de statistique pour insiter à offrir du stockage, du cpu, des annuaires (bonus) + + +Contacts +-------- +- Gerson Sunye pour le DHT +- Lamarre pour le plus haut niveau (les algos de recherche/decouverte...) +- TMG pour le P2P (on va avoir les mêmes problématiques : connexion entrante sur un pair qui n'est pas un serveur) + + +Technique +--------- +- JXTA : protocole P2P +- rechercher s'il n'y a pas d'autre librairie utilisable. Par exemple +regarder dans les implantations java bittorrent s'il n'y a pas des +choses recuperables (vuze, ...) + + +Premier objectif +---------------- +Bonne structure de l'annuaire. +Etude des protocoles. +Solutionnement des problématiques de communication (firewall). + +Fonctionne sans réplication des données. +Fonctionne sans téléchargement depuis plusieurs sources. + +La version 1.0.0 doit être entierement fonctionnelle (sans contenir toutes les +fonctionnalités) \ No newline at end of file Deleted: trunk/src/site/rst/diswork-daemon/how_it_works.rst =================================================================== --- trunk/src/site/rst/diswork-daemon/how_it_works.rst 2010-06-29 11:13:09 UTC (rev 92) +++ trunk/src/site/rst/diswork-daemon/how_it_works.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -1,91 +0,0 @@ -======================== -How Diswork Daemon works -======================== - -Overview --------- - -Diswork Daemon uses Diswork File System. - -Submit a job ------------- - -First, a node, before joining a network generate, for himself, a unique ID. And -then, create a directory "/home/ID". If this is not the first time the node -is run, ID should be reused so jobs can be found back after disconnect/connect. - -In his home directory, a node can put and remove his jobs. - -To create a job, a node should : - -#. Check that the application in the good version is available at - /bin/application/application-version.jar - -#. Create a directory in home for each of his jobs - /home/ID/job-1, /home/ID/job-2... - -#. In a job directory, put job.jsdl, a command line to run. This file contains - necessary information like : - - * The name of the job - * The application name, version - * File staging (needed file and where to get them) - * ... (to be completed) - -#. Declare the job as available by adding a symlink in /var/jobs/TODO - with the link named after the current date, the id of the job owner and - the job name. Places possible in /var/jobs/ are : - -#. Check the content of /home/ID/job-i/status - -#. Finally, clean home by deleting the directories of finished jobs - -Perform a job -------------- - -To perform a job, a node should find a job in /var/jobs/, this directory has -6 sub-directories : - -- /var/jobs/TODO - -- /var/jobs/FAILED_1 - -- /var/jobs/FAILED_2 - -- /var/jobs/FAILED_3 - -- /var/jobs/DONE - -TODO, FAILED_1, FAILED_2 all have a sub-directory RUNNING. - -#. Choose a job in one of those directories except DONE and FAILED_3. Use the - data in the link same to know which job should be done first; - -#. Once the job is chosen, move the symlink to the RUNNING subdir where you - found the link. Then, follow the link to get the actual datas needed for - the job; - -#. Run the job; - -#. Now there is two way to proceed : - - - If the job successfully ran (no exception raised and return code is 0) : - - #. write the results in the job directory; - - #. move the link to DONE. - - - If the job has failed (a job taking too much time may be considered as - a failure). move it to according to where the job was found. If - the job was in TODO, move it to FAILED_1 (read "failed once") ; - if the job was in FAILED_1, move it to FAILED_2, if it was it FAILED_2, - move it to FAILED_3. - - In both cases, write what occurred in the job directory in the status.txt - file (append your message, don't remove anything); - -#. Sometimes, while choosing a job, check in the RUNNING directory if any - symlink is not too old (use the name of the link). It may means that a - node crashed while processing this job. Move it to the RUNNING directory - of the next failure level (TODO → FAILED_1 → FAILED_2 → FAILED_3) and try - process it. \ No newline at end of file Copied: trunk/src/site/rst/diswork-fs/devel.rst (from rev 92, trunk/src/site/rst/diswork-fs/how_it_works.rst) =================================================================== --- trunk/src/site/rst/diswork-fs/devel.rst (rev 0) +++ trunk/src/site/rst/diswork-fs/devel.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -0,0 +1,148 @@ +============================= +How Diswork File System works +============================= + +Abstract +-------- + +This document is for people who want to develop Diswork File System. It is not +an interesting reading for people who write to write code that use Diswork +FS. + +Diswork File System uses a map to store every data +-------------------------------------------------- + +Diswork File System uses a map to store data persistently. + +We illustrate the use of a file system by setting the content of "/", +the root directory as follow: + +* a directory named "my_folder" that contains: + + - a directory named "my_sub_folder" that contains: + + + a file name "my_deep_file.ext" + + + an empty folder named "my_empty_folder" + + - a file named "my_other_file.ext" + +* a file named "my_file.ext" (containing binary data: 010101) + +* a symbolic link named "my_link" which target the path + "/my_folder/my_sub_folder/my_deep_file.ext" + +Storage uses a Map to store data. + ++--------------+---------------------------------------------+ +| Key (String) | Value (byte[]) | ++==============+=============================================+ +| | "D:my_folder:ID1\n | +| | F:my_file.ext:ID2\n | +| "/" | L:my_link:ID3" | ++--------------+---------------------------------------------+ +| | "D:my_sub_folder:ID4\n | +| "ID1" | F:my_other_file.ext:ID5" | ++--------------+---------------------------------------------+ +| "ID2" | 010101 | ++--------------+---------------------------------------------+ +| "ID3" | "/my_folder/my_sub_folder/my_deep_file.ext" | ++--------------+---------------------------------------------+ +| | "F:my_deep_file:ID6\n | +| "ID4" | D:my_empty_folder:ID7" | ++--------------+---------------------------------------------+ +| "ID5" | 011010100111010... | ++--------------+---------------------------------------------+ +| "ID6" | 1100010101010110110010... | ++--------------+---------------------------------------------+ +| "ID7" | "" | ++--------------+---------------------------------------------+ + +An entry is, for example, "D:my_folder:ID1". It contains the type of the +entry (<strong>D</strong>irectory, <strong>F</strong>ile or +<strong>L</strong>ink), the name of the element, and an ID to be used as +a key on the map to get the actual content. Those three informations are +separated by ":" which is EntryUtil.ENTRY_SEPARATOR. + +A directory way have multiple <em>entries</em>. Entries of a directory are +separated by "\n" which is EntryUtil.ENTRIES_SEPARATOR. + +The above description shows the main principle used to store a tree +structure in a map. + +In fact, the storage of actual data is a bit more complex, due to +the need of splitting the data that may be too large (the value of "ID6" +may be a series of 1002<sup>20</sup> bytes for a 200 MiB file). +Actually, files content (like ID2, ID5, ID6) and directory content +values ("/", ID1, ID4) will be split and the map will look like that ( +<strong>note that only the files have been split for readability</strong>): + ++--------------+---------------------------------------------+ +| Key (String) | Value (byte[]) | ++==============+=============================================+ +| | "D:my_folder:ID1\n | +| | F:my_file.ext:ID2\n | +| "/" | L:my_link:ID3" | ++--------------+---------------------------------------------+ +| | "D:my_sub_folder:ID4\n | +| "ID1" | F:my_other_file.ext:ID5" | ++--------------+---------------------------------------------+ +| "ID2" | "6;ID8" | ++--------------+---------------------------------------------+ +| "ID3" | "/my_folder/my_sub_folder/my_deep_file.ext" | ++--------------+---------------------------------------------+ +| | "F:my_deep_file:ID6\n | +| "ID4" | D:my_empty_folder:ID7" | ++--------------+---------------------------------------------+ +| "ID5" | "14689;ID9" | ++--------------+---------------------------------------------+ +| "ID6" | "79874567;ID10;ID11;ID12" | ++--------------+---------------------------------------------+ +| "ID7" | "" | ++--------------+---------------------------------------------+ +| "ID8" | 010101 | ++--------------+---------------------------------------------+ +| "ID9" | 011010100111010... | ++--------------+---------------------------------------------+ +| "ID10" | 110001010101011... | ++--------------+---------------------------------------------+ +| "ID11" | 111010110101101... | ++--------------+---------------------------------------------+ +| "ID12" | 011101100111010... | ++--------------+---------------------------------------------+ + +Values of ID2, ID5, ID6 are no longer the file content. It's now a set +of meta-information information data called metablock. +A metablock is composed of the total size of the file followed by +an ordered lists of IDs of the different blocks composing the file. +Those informations are separated by ";" (see +{@link org.nuiton.disworkfs.storage.EntryUtil#BLOCKIDS_SEPARATOR}). + +In the above example: + +* "/my_file.ext" is a file which size is 6 bytes, it is + composed of one block available at ID8; + +* "/my_sub_folder/my_other_file.ext" is a file (size = 14689 bytes) + which content is available at ID9; + +* "/my_sub_folder/my_sub_folder/my_deep_file.ext" is a big file: its + size is 79874567. It has been split in three blocks: ID10, ID11 and + ID12. + +When reading and writing in storage, split is done transparently. When +reading, a Stream is returned: it loads data blocks after blocks +when needed inner class Storage.SplitBlocksInputStream}). +When writing, data are split in blocks of a maximum configurable size. + +The map is distributed +---------------------- + +The map described above is distributed over all nodes through a DHT. + +Diswork File System manage concurrency +-------------------------------------- + +DFS uses copy-on-write to deal with concurrency. Nothing is overridden, when +a file is replaced, the new file is written on the FS without removing the +old file. The old file data is kept and actually removed later. \ No newline at end of file Deleted: trunk/src/site/rst/diswork-fs/how_it_works.rst =================================================================== --- trunk/src/site/rst/diswork-fs/how_it_works.rst 2010-06-29 11:13:09 UTC (rev 92) +++ trunk/src/site/rst/diswork-fs/how_it_works.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -1,136 +0,0 @@ -============================= -How Diswork File System works -============================= - -Diswork File System uses a map to store every data --------------------------------------------------- - -Diswork File System uses a map to store data persistently. - -We illustrate the use of a file system by setting the content of "/", -the root directory as follow: - -* a directory named "my_folder" that contains: - - - a directory named "my_sub_folder" that contains: - - + a file name "my_deep_file.ext" - - + an empty folder named "my_empty_folder" - - - a file named "my_other_file.ext" - -* a file named "my_file.ext" (containing binary data: 010101) - -* a symbolic link named "my_link" which target the path - "/my_folder/my_sub_folder/my_deep_file.ext" - -Storage uses a Map to store data. - -+--------------+---------------------------------------------+ -| Key (String) | Value (byte[]) | -+==============+=============================================+ -| | "D:my_folder:ID1\n | -| | F:my_file.ext:ID2\n | -| "/" | L:my_link:ID3" | -+--------------+---------------------------------------------+ -| | "D:my_sub_folder:ID4\n | -| "ID1" | F:my_other_file.ext:ID5" | -+--------------+---------------------------------------------+ -| "ID2" | 010101 | -+--------------+---------------------------------------------+ -| "ID3" | "/my_folder/my_sub_folder/my_deep_file.ext" | -+--------------+---------------------------------------------+ -| | "F:my_deep_file:ID6\n | -| "ID4" | D:my_empty_folder:ID7" | -+--------------+---------------------------------------------+ -| "ID5" | 011010100111010... | -+--------------+---------------------------------------------+ -| "ID6" | 1100010101010110110010... | -+--------------+---------------------------------------------+ -| "ID7" | "" | -+--------------+---------------------------------------------+ - -An entry is, for example, "D:my_folder:ID1". It contains the type of the -entry (<strong>D</strong>irectory, <strong>F</strong>ile or -<strong>L</strong>ink), the name of the element, and an ID to be used as -a key on the map to get the actual content. Those three informations are -separated by ":" which is EntryUtil.ENTRY_SEPARATOR. - -A directory way have multiple <em>entries</em>. Entries of a directory are -separated by "\n" which is EntryUtil.ENTRIES_SEPARATOR. - -The above description shows the main principle used to store a tree -structure in a map. - -In fact, the storage of actual data is a bit more complex, due to -the need of splitting the data that may be too large (the value of "ID6" -may be a series of 1002<sup>20</sup> bytes for a 200 MiB file). -Actually, files content (like ID2, ID5, ID6) and directory content -values ("/", ID1, ID4) will be split and the map will look like that ( -<strong>note that only the files have been split for readability</strong>): - -+--------------+---------------------------------------------+ -| Key (String) | Value (byte[]) | -+==============+=============================================+ -| | "D:my_folder:ID1\n | -| | F:my_file.ext:ID2\n | -| "/" | L:my_link:ID3" | -+--------------+---------------------------------------------+ -| | "D:my_sub_folder:ID4\n | -| "ID1" | F:my_other_file.ext:ID5" | -+--------------+---------------------------------------------+ -| "ID2" | "6;ID8" | -+--------------+---------------------------------------------+ -| "ID3" | "/my_folder/my_sub_folder/my_deep_file.ext" | -+--------------+---------------------------------------------+ -| | "F:my_deep_file:ID6\n | -| "ID4" | D:my_empty_folder:ID7" | -+--------------+---------------------------------------------+ -| "ID5" | "14689;ID9" | -+--------------+---------------------------------------------+ -| "ID6" | "79874567;ID10;ID11;ID12" | -+--------------+---------------------------------------------+ -| "ID7" | "" | -+--------------+---------------------------------------------+ -| "ID8" | 010101 | -+--------------+---------------------------------------------+ -| "ID9" | 011010100111010... | -+--------------+---------------------------------------------+ -| "ID10" | 110001010101011... | -+--------------+---------------------------------------------+ -| "ID11" | 111010110101101... | -+--------------+---------------------------------------------+ -| "ID12" | 011101100111010... | -+--------------+---------------------------------------------+ - -Values of ID2, ID5, ID6 are no longer the file content. It's now a set -of meta-information information data called metablock. -A metablock is composed of the total size of the file followed by -an ordered lists of IDs of the different blocks composing the file. -Those informations are separated by ";" (see -{@link org.nuiton.disworkfs.storage.EntryUtil#BLOCKIDS_SEPARATOR}). - -In the above example: - -* "/my_file.ext" is a file which size is 6 bytes, it is - composed of one block available at ID8; - -* "/my_sub_folder/my_other_file.ext" is a file (size = 14689 bytes) - which content is available at ID9; - -* "/my_sub_folder/my_sub_folder/my_deep_file.ext" is a big file: its - size is 79874567. It has been split in three blocks: ID10, ID11 and - ID12. - -When reading and writing in storage, split is done transparently. When -reading, a Stream is returned: it loads data blocks after blocks -when needed inner class Storage.SplitBlocksInputStream}). -When writing, data are split in blocks of a maximum configurable size. - -Diswork File System manage concurrency --------------------------------------- - -DFS uses copy-on-write to deal with concurrency. Nothing is overridden, when -a file is replaced, the new file is written on the FS without removing the -old file. The old file data is kept and actually removed later. \ No newline at end of file Added: trunk/src/site/rst/user/how_to_use.rst =================================================================== --- trunk/src/site/rst/user/how_to_use.rst (rev 0) +++ trunk/src/site/rst/user/how_to_use.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -0,0 +1,15 @@ +================== +How to use Diswork +================== + +How to install a diswork node on my computer +-------------------------------------------- + +How to install a diswork node on my server +------------------------------------------ + +How to make my own application ready for being run by all diswork nodes +----------------------------------------------------------------------- + +How to make my application able to submit jobs to Diswork and retrieve results +------------------------------------------------------------------------------ \ No newline at end of file Deleted: trunk/src/site/rst/user/install.rst =================================================================== --- trunk/src/site/rst/user/install.rst 2010-06-29 11:13:09 UTC (rev 92) +++ trunk/src/site/rst/user/install.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -1,5 +0,0 @@ -Install -======= - -TODO: client use -TODO: server use \ No newline at end of file Deleted: trunk/src/site/rst/user/starting.rst =================================================================== --- trunk/src/site/rst/user/starting.rst 2010-06-29 11:13:09 UTC (rev 92) +++ trunk/src/site/rst/user/starting.rst 2010-07-01 10:38:20 UTC (rev 93) @@ -1,4 +0,0 @@ -Starting with Diswork -===================== - -TODO : maybe put here a "hello world" application running on diswork \ No newline at end of file Modified: trunk/src/site/site_en.xml =================================================================== --- trunk/src/site/site_en.xml 2010-06-29 11:13:09 UTC (rev 92) +++ trunk/src/site/site_en.xml 2010-07-01 10:38:20 UTC (rev 93) @@ -19,21 +19,16 @@ </breadcrumbs> <menu name="Users"> - <item href="user/install.html" name="Install"/> - <item href="user/starting.html" name="Starting"/> + <item href="user/how_to_use.html" name="How to use"/> </menu> - <menu name="Developer"> - <item href="devel/draft.html" name="First draft"/> - </menu> - <menu name="diswork daemon"> - <item href="diswork-daemon/how_it_works.html" name="How it works"/> + <item href="diswork-daemon/devel.html" name="For developers"/> </menu> <menu name="diswork FS"> <item href="diswork-fs/history.html" name="History of the project"/> - <item href="diswork-fs/how_it_works.html" name="How it works"/> + <item href="diswork-fs/devel.html" name="For developers"/> </menu> <menu ref="reports"/> Added: trunk/src/uml/diswork.zargo =================================================================== (Binary files differ) Property changes on: trunk/src/uml/diswork.zargo ___________________________________________________________________ Added: svn:mime-type + application/octet-stream