[scire] Log of /server

Revision 247 - Directory Listing
Modified Tue Aug 7 01:36:04 2007 UTC (5 years, 9 months ago) by codeman

This is a very large commit from Rodrigo Lazo (rlazo) for Google SoC
This is the latest patch on the job execution series. As far as I
could test it, now scire should be able to create, distribute, execute
and gather the output of a job with the new models.

Also the jobs register their status on the database (pending,
downloaded, running, failed, finished or cancelled). So now all the
backend code needed to "watch" a job is on its place.

Right now a job gets executed but dies on the post processing stage
(at least that is what I understand from the error message)

Just move the GACL_functions.py from server/ to server/modules and
apply this patch and you'll how myltiple entries are created for each
member of the group in jobs_clients and on job_history.

I've take some precautions regarding concurrency adding some locks,
I'm not sure they are enough so please comment on that

Revision 244 - Directory Listing
Modified Wed Aug 1 03:03:41 2007 UTC (5 years, 9 months ago) by codeman

patch from rlazo for SoC
fixing statusid

Revision 243 - Directory Listing
Modified Wed Aug 1 00:58:49 2007 UTC (5 years, 9 months ago) by codeman

patch from Rodrigo Lazo for Google SoC:
Fixed-a-minor-error-with-gacl_axo_seq-handling

Revision 234 - Directory Listing
Modified Sat Jul 21 03:38:27 2007 UTC (5 years, 10 months ago) by codeman

adding back in the sequences DB code
adding in some code that will hopefully allow clients to
once-again be inserted.

Revision 229 - Directory Listing
Modified Thu Jul 19 01:19:36 2007 UTC (5 years, 10 months ago) by codeman

adding a comment from rlazo

Revision 227 - Directory Listing
Modified Fri Jul 6 16:18:12 2007 UTC (5 years, 10 months ago) by codeman

adding two cron-handling files from rlazo for Google SoC

Revision 225 - Directory Listing
Modified Sat Jun 16 18:13:01 2007 UTC (5 years, 11 months ago) by codeman

making some changes from Rodrigo Lazo for SoC:
Also, the
job requesting mechanism uses the new run scheduling model
(crontab-like).

it looks like the basic structure has been prepared for this
code to be inserted later.  the old stuff has been removed.

Revision 224 - Directory Listing
Modified Sat Jun 16 18:07:02 2007 UTC (5 years, 11 months ago) by codeman

updates from Rodrigo Lazo for Google SoC:
Fixed the backend and UI code so it can work with the latest
modification to the db (client_status, jobs_status tables). Also, the
job requesting mechanism uses the new run scheduling model
(crontab-like).

Revision 217 - Directory Listing
Modified Thu Jun 7 03:51:45 2007 UTC (5 years, 11 months ago) by codeman

my bad.
bind address actually should default to 0.0.0.0

Revision 216 - Directory Listing
Modified Thu Jun 7 03:01:58 2007 UTC (5 years, 11 months ago) by codeman

fix messed up error string.

Revision 214 - Directory Listing
Modified Thu Jun 7 02:29:06 2007 UTC (5 years, 11 months ago) by codeman

changed the tmpdir to /etc/scire
changed the default bind_address to localhost.

Added some code from rlazo (Rodrigo Lazo) for SoC:
this adds command-line args for debugging, host, port, and config dir.

Revision 213 - Directory Listing
Modified Thu Jun 7 00:30:31 2007 UTC (5 years, 11 months ago) by codeman

re-adding the postprocessing and db_functions files
they have changed a bit since their deleted versions.

Revision 212 - Directory Listing
Modified Thu Jun 7 00:29:46 2007 UTC (5 years, 11 months ago) by codeman

moving db_functions and postprocess elsewhere.
adding if debug: for a more verbose setting to be allowed
fixing up the postprocessing code to correctly import
the file and execute.  this will become a scan
of the directory in the future.

Revision 210 - Directory Listing
Modified Mon Jun 4 02:04:33 2007 UTC (5 years, 11 months ago) by codeman

just trying to make this work again

Revision 209 - Directory Listing
Modified Mon Jun 4 02:04:15 2007 UTC (5 years, 11 months ago) by codeman

some touchups to keep it compiling.

Revision 208 - Directory Listing
Modified Thu May 31 03:54:54 2007 UTC (5 years, 11 months ago) by codeman

tossing in two files to deal with post_processing.
they need a LOT of work.

Revision 207 - Directory Listing
Modified Thu May 31 03:53:41 2007 UTC (5 years, 11 months ago) by codeman

fixing up the pseudo-code a bit so it may actually be runnable

Revision 201 - Directory Listing
Modified Mon May 21 04:03:34 2007 UTC (6 years ago) by codeman

adding sample postprocess code.  this will clearly NOT work, but
it's just an example to show the theory behind how it might work.

Revision 189 - Directory Listing
Modified Sun Mar 25 21:35:30 2007 UTC (6 years, 2 months ago) by mdisney

Just removed some comments cruft from the server modules.

Revision 178 - Directory Listing
Modified Mon Dec 11 00:57:18 2006 UTC (6 years, 5 months ago) by codeman

some bugfixes and debugging added/uncommented.
origjob is causing some issues b/c it doesn't exist
in the DB schema yet.
added semi-proper cleanup after job completion
added line to repr(output) so xmlrpclib doesn't barf

Revision 167 - Directory Listing
Modified Sun Dec 10 04:49:36 2006 UTC (6 years, 5 months ago) by agaffney

  server/SecureXMLRPCServer.py:
  switch to ForkingMixIn instead of ThreadingMixIn to prevent cert info from               
  bleeding between connection instances.
  reverse order of inherited classes so MixIn works properly

Revision 166 - Directory Listing
Modified Sun Dec 10 01:20:42 2006 UTC (6 years, 5 months ago) by codeman

bugfixes on origjob mostly.
added an update to the pending status of the job.

Revision 152 - Directory Listing
Modified Sat Aug 26 15:35:01 2006 UTC (6 years, 8 months ago) by codeman

Adding in large changes from mdisney.
including his complete scire svn logs from the soc svn repository
so they don't get lost.
------------------------------------------------------------------------
r28 | mdisney | 2006-08-20 15:35:40 -0400 (Sun, 20 Aug 2006) | 4 lines

- Add logic to get_jstatus and job_cancelled to account
  for recurring jobs. This should complete full support
  in the backend for recurring jobs.

------------------------------------------------------------------------
r27 | mdisney | 2006-08-20 12:54:26 -0400 (Sun, 20 Aug 2006) | 3 lines

- Added functionality to get_jobs that checks to ensure the
  run_interval has passed before adding a job to the list.

------------------------------------------------------------------------
r26 | mdisney | 2006-08-19 23:28:42 -0400 (Sat, 19 Aug 2006) | 10 lines

- This commit introduces an approach to recurring jobs that
  duplicates any recurring job into its own row when retrieved
  by get_job. Job retrieval is currently working though get_jstatus
  and job_cancelled need some work so that they can deal with
  the exception of recurring jobs. There may be other functions
  that need work as well.

- Minor: commented out some unrelated debug print statements.


------------------------------------------------------------------------
r25 | mdisney | 2006-08-17 19:02:55 -0400 (Thu, 17 Aug 2006) | 23 lines

- scirec now accepts -D for daemon mode. Much of the "main"
  code has been moved into scirec_main, which is called in a while-true
  loop in daemon mode. The polling interval is set at the top of the
  file.  This crude while-true loop is probably not the best solution
  for a continuous poll/sleep but putting something like this in place
  was necessary to proceed with job cancellations. Without some sort
  of continuous polling, unless we spool out our jobs to a file or
  client-side database, job cancellation doesn't make sense.

- Added logic to run_job that will, before checking for dependencies,
  make a call to the server to see whether the job has been cancelled.
  If so, the job is not run. The history is recorded as "Aborted".

- To support job cancelling, a subroutine was added on the server:
  job_cancelled. This will look for Cmd:Abort in the job_history
  table. That string represents "The command to abort this job has
  been issued." Perhaps there will be applications for other "Cmd:"
  uses. This "Cmd:" business is to distinguish it from the past-tense
  state of "Aborted".

- Fixed a bug where I think the jobs list was not keeping the known
  jobs on the client.

------------------------------------------------------------------------
r24 | mdisney | 2006-08-15 22:10:00 -0400 (Tue, 15 Aug 2006) | 2 lines

- Removed the parsing for different dependency types.

------------------------------------------------------------------------
r23 | mdisney | 2006-08-15 21:01:52 -0400 (Tue, 15 Aug 2006) | 24 lines

- Summary: Basic ependency checking should now work. The job_dependency
  field should be a string of potentially multiple jobid, separated
  by commas and no whitespace. Right now the characters 'a' or 'w'
  may be placed at the end of a jobid in the job_dependency field
  to represent a job that should be autoexecuted or one that is
  designated to be waited on for completion. While these different
  types are parsed out, they are all handled the same currently:
  if the status query is not 'Succeeded' then the job does not get
  run and it gets recorded in the job_history table with a special
  eventmsg describing a dependency failure and which jobid it was.

- Added the get_jstatus subroutine in jobs.py which
  will query the job_history table and return the job status
  for a certain combination of both jobid and clientid.

- The get_job subroutine now also returns job_condition info.

- Changed get_job to cast every key in the job dict to be a
  string to avoid marshalling problems, at least for a while.

- The dependency checking occurs in the run_job subroutine on the
  client and depends on get_jstatus.


------------------------------------------------------------------------
r22 | mdisney | 2006-08-10 17:39:42 -0400 (Thu, 10 Aug 2006) | 11 lines

- Fixed the marshalling error, which was due to not returning any
  value at all in job_return.
- Recording of job_history now works as follows:
  1. If the job was successful (meaning that the return code
     matches the scripts.success_code), "Succeeded" will be recorded
     as the status in the job_history table. No eventmsg will be
     recorded.
  2. If the job was not successful, "Failed" will be the status
     in job_history and the eventmsg will be recorded (eventmsg is
     currently a varchar(255)).

------------------------------------------------------------------------
r21 | mdisney | 2006-08-08 18:18:13 -0400 (Tue, 08 Aug 2006) | 9 lines

- Added syslog logging of job output.
- Improved the logging of job output to file, using the "logging"
  module. Logging to file will happen in addition to syslog if the
  logfile for the script is defined in the database.
- Started straightening out the returning of job results to the
  server. Created a job_return() function on the server, but there
  is currently a bug resulting in one of my old favorite xml-rpc
  marshalling of "None" errors.

------------------------------------------------------------------------
r20 | mdisney | 2006-08-06 18:29:22 -0400 (Sun, 06 Aug 2006) | 3 lines

- Moved the conditionals for the start/end period into the
  SELECT.

------------------------------------------------------------------------
r19 | mdisney | 2006-08-06 17:24:51 -0400 (Sun, 06 Aug 2006) | 6 lines

Merging in changes from primary repo, including:
- Fixes for client status (e.g. Active not approved)
- Fix for SQL statement in get_jobs
- More debugging info
- Fix for xmlrpclib datetime problem

------------------------------------------------------------------------
r18 | mdisney | 2006-08-03 22:26:23 -0400 (Thu, 03 Aug 2006) | 3 lines

- start/end_period job restraint in progress. This rev has
  problems with datetime marshalling.

------------------------------------------------------------------------
r17 | mdisney | 2006-08-03 20:08:22 -0400 (Thu, 03 Aug 2006) | 6 lines

- Added a LEFT JOIN to the sql in get_jobs that should get
  us the info we need from job_conditions to do job
  scheduling. Needed to use a LEFT join because if we used
  a regular one, zero entries in job_conditions for a job
  would result in zero jobs getting returned.

------------------------------------------------------------------------
r16 | mdisney | 2006-08-03 15:53:33 -0400 (Thu, 03 Aug 2006) | 3 lines

- Previous commit did not include the condition for
  add_client or register_client in _dispatch.

------------------------------------------------------------------------
r15 | mdisney | 2006-08-03 15:43:11 -0400 (Thu, 03 Aug 2006) | 3 lines

- Added client registration check in _dispatch, including
  modifying register_client to return client status.

------------------------------------------------------------------------
r14 | mdisney | 2006-08-03 12:40:30 -0400 (Thu, 03 Aug 2006) | 5 lines

- Now executing the script with getstatusoutput.
- Fixed up logging just a bit. Log file was being overwritten,
  not appended to. And now each execution gets a timestamp in
  the log.

------------------------------------------------------------------------
r13 | mdisney | 2006-08-03 11:39:25 -0400 (Thu, 03 Aug 2006) | 3 lines

- Small change to get_jobs that sorts the jobs list based on
  priority then created date (simply using database "order by").

------------------------------------------------------------------------
r12 | mdisney | 2006-08-02 17:56:08 -0400 (Wed, 02 Aug 2006) | 3 lines

- Added gen_summary and gen_job_digest functions.
- Implemented job digest comparison in get_jobs.

------------------------------------------------------------------------
r11 | mdisney | 2006-08-02 14:08:27 -0400 (Wed, 02 Aug 2006) | 4 lines

- get_jobs() now performs a db query against jobs_clients.
  Comparison against provided array of hashes of jobs not
  yet functional.

------------------------------------------------------------------------
r10 | mdisney | 2006-07-25 16:51:23 -0400 (Tue, 25 Jul 2006) | 3 lines

- Changed self -> client_digest in server/modules/client.py, and
  made appropriate changes in scirec.py

------------------------------------------------------------------------
r9 | mdisney | 2006-07-25 16:41:25 -0400 (Tue, 25 Jul 2006) | 6 lines

- Client now gets added if doesn't already exist in the db.
- Changed the cert passing for add_client to actually pass the
  text of the cert itself, which then gets loaded on the server
  side.
- Corrected some problems with SQL in add_client().

------------------------------------------------------------------------
r8 | mdisney | 2006-07-24 17:46:00 -0400 (Mon, 24 Jul 2006) | 6 lines

- Added some debugging to scired and scirec.
- Fixed a syntax problem server/modules/client.py
- Other forward movement on add_client(). Current error
  is related to marshalling X509 objects.


------------------------------------------------------------------------
r7 | mdisney | 2006-07-24 16:15:36 -0400 (Mon, 24 Jul 2006) | 9 lines

- Added functionality for calling register_client(client_digest) to
  scirec.py.
- Changed SQL calls in /server/modules/client.py to use parameter
  substitution in the Execute call itself.
- Created add_client in /server/modules/client.py, but having a
  problem in that scired will now not register the module.



------------------------------------------------------------------------
r6 | mdisney | 2006-07-24 08:26:13 -0400 (Mon, 24 Jul 2006) | 3 lines

Clean up of sys_info() by agaffney.
Moved add_client function into modules/client.py.

------------------------------------------------------------------------
r5 | mdisney | 2006-07-23 16:58:09 -0400 (Sun, 23 Jul 2006) | 7 lines

Implemented sys_info() on the client side, which will collect
the system info that will be passed to the method for adding
clients to populate the database table.

Currently still experiencing difficulty with calling the
add_client method.

------------------------------------------------------------------------
r4 | mdisney | 2006-07-23 12:17:27 -0400 (Sun, 23 Jul 2006) | 1 line

removed old original file used for creating patches against a certain release
------------------------------------------------------------------------
r3 | mdisney | 2006-07-23 12:16:05 -0400 (Sun, 23 Jul 2006) | 1 line

removed pyc files
------------------------------------------------------------------------
r2 | mdisney | 2006-07-23 12:08:10 -0400 (Sun, 23 Jul 2006) | 1 line

Initial import with clients table stuff in progress.
------------------------------------------------------------------------
r1 | mdisney | 2006-07-23 11:51:49 -0400 (Sun, 23 Jul 2006) | 1 line

Revision 149 - Directory Listing
Modified Sat Aug 5 23:42:07 2006 UTC (6 years, 9 months ago) by codeman

clean up datetime bug and get_jobs query a bit

Revision 147 - Directory Listing
Modified Sat Aug 5 22:22:09 2006 UTC (6 years, 9 months ago) by codeman

some little changes of my own.
changed pending to Pending and accepted to
Active for consistancy with FE

Revision 145 - Directory Listing
Modified Sat Aug 5 21:06:28 2006 UTC (6 years, 9 months ago) by codeman

Lots of updates from Matt Disney (mdisney)
see his svn for more thorough logs.

Revision 143 - Directory Listing
Modified Mon Jul 24 00:07:36 2006 UTC (6 years, 10 months ago) by agaffney

move add_client function to the client module

Revision 141 - Directory Listing
Modified Sun Jul 23 21:25:15 2006 UTC (6 years, 10 months ago) by codeman

updates from mdisney:
  added a client function called sys_info() which basically populates a small dictionary with system information which will be passed to a method on the 
server to register/add the client. Then the system information will be added into the database along with the certificate information.
  added the beginnings of the code to add/register the client in scired.py

Revision 135 - Directory Listing
Modified Sun Jul 23 01:00:51 2006 UTC (6 years, 10 months ago) by agaffney

add local traceback output on error for xmlrpc functions

Revision 134 - Directory Listing
Modified Sun Jul 23 01:00:10 2006 UTC (6 years, 10 months ago) by codeman

updating with associative fetches for job.py
this is much cleaner code and allows for prepared statements later.

Revision 132 - Directory Listing
Modified Sun Jul 23 00:42:36 2006 UTC (6 years, 10 months ago) by agaffney

kill stray 's' on unspool_job

Revision 131 - Directory Listing
Modified Sun Jul 23 00:11:30 2006 UTC (6 years, 10 months ago) by agaffney

added module reloading and moved job-related code into modules/joy.py

Revision 129 - Directory Listing
Modified Sat Jul 22 23:32:44 2006 UTC (6 years, 10 months ago) by agaffney

initial plugin arch stuff

Revision 127 - Directory Listing
Modified Sat Jul 15 05:02:19 2006 UTC (6 years, 10 months ago) by codeman

Updating the scire server and client programs with their current
development copy.  All changes from previous version have been
made by Matt Disney as part of the Summer of Code.  These are
somewhat large updates involving database connectivity using
python's ADOdb module and function calls between the server and
client.  Many thanks to Matt for the contribution.

Revision 126 - Directory Listing
Modified Sat Jul 15 04:56:24 2006 UTC (6 years, 10 months ago) by codeman

Adding a Python-port of PHP-GACL's acl_check function,
written by Brian Coca.  This is submitted with his permission.
this will take an ADOdb object (or config info), and then return
the result of acl_check() queries as in PHP.  Many thanks to
Brian for the port

Revision 117 - Directory Listing
Modified Mon Jun 19 15:49:32 2006 UTC (6 years, 11 months ago) by agaffney

don't print cert digest in client

Revision 116 - Directory Listing
Modified Mon Jun 19 15:24:25 2006 UTC (6 years, 11 months ago) by agaffney

.svnignore sucks

Revision 115 - Directory Listing
Modified Mon Jun 19 15:21:54 2006 UTC (6 years, 11 months ago) by agaffney

add .svnignore

Revision 114 - Directory Listing
Modified Mon Jun 19 15:18:56 2006 UTC (6 years, 11 months ago) by agaffney

remove .svnignore files

Revision 113 - Directory Listing
Modified Mon Jun 19 15:16:09 2006 UTC (6 years, 11 months ago) by agaffney

add .svnignore files

Revision 107 - Directory Listing
Modified Mon Jun 19 01:24:37 2006 UTC (6 years, 11 months ago) by agaffney

add client cert verification to server

Revision 101 - Directory Listing
Modified Thu Jun 15 12:22:43 2006 UTC (6 years, 11 months ago) by agaffney

update of new xmlrpc classes

Revision 76 - Directory Listing
Modified Sun May 21 04:48:42 2006 UTC (7 years ago) by agaffney

move cert generation to certgen.py

Revision 63 - Directory Listing
Modified Fri May 19 03:14:50 2006 UTC (7 years ago) by agaffney

SSL-enabled server and basic client

Revision 62 - Directory Listing
Modified Fri May 19 00:41:53 2006 UTC (7 years ago) by agaffney

move blackace's implementation into a subdir

Revision 60 - Directory Listing
Modified Thu May 18 13:46:12 2006 UTC (7 years ago) by blackace

Add scireserver.py and two classes it uses.

Revision 56 - Directory Listing
Added Mon May 8 03:19:46 2006 UTC (7 years ago) by agaffney

initial commit of scired

(Current path doesn't exist after revision 247)

	ViewVC Help
Powered by ViewVC 1.1.13