gstats.git
6 years agoAdd support for pkgcore to the client; trivial uploads now supported by master
Alec Warner [Sun, 18 May 2008 12:10:18 +0000 (05:10 -0700)]
Add support for pkgcore to the client; trivial uploads now supported by
both managers.

Extract out the USE flag simplification since it is common.

6 years agoFix bug in gstats-client.
Alec Warner [Sun, 18 May 2008 09:41:47 +0000 (02:41 -0700)]
Fix bug in gstats-client.

Fix up gstats_storage to handle xml files, 128-bit random hostIDs.

fix gstats_processor to inject this new xml format (still needs work I think).

6 years agoAdd function for 500 error page. add HostAdd.
Alec Warner [Sun, 18 May 2008 09:07:54 +0000 (02:07 -0700)]
Add function for 500 error page.  add HostAdd.

6 years agoAdd beginnins of host API, remove cpv API.
Alec Warner [Sun, 18 May 2008 05:39:24 +0000 (22:39 -0700)]
Add beginnins of host API, remove cpv API.

add more 404 handlers.

6 years agoif there are no use flags omit the <use></use> xml.
Alec Warner [Sun, 18 May 2008 05:04:23 +0000 (22:04 -0700)]
if there are no use flags omit the <use></use> xml.

Also add <packages></packages> header + footer

6 years agooops, use sets for intersection ;)
Alec Warner [Sun, 18 May 2008 05:01:24 +0000 (22:01 -0700)]
oops, use sets for intersection ;)

6 years agoTurns out USE is funky and generates tons of extra data we don't care about (USE_EXPA...
Alec Warner [Sun, 18 May 2008 05:00:26 +0000 (22:00 -0700)]
Turns out USE is funky and generates tons of extra data we don't care about (USE_EXPAND mostly).

Nuke that crap by doign teh IUSE USE intersection

6 years agoIndent XML
Alec Warner [Sun, 18 May 2008 04:55:01 +0000 (21:55 -0700)]
Indent XML

6 years agousing set() around keys fucks with their order and ruins the zip() call later.
Alec Warner [Sun, 18 May 2008 04:54:07 +0000 (21:54 -0700)]
using set() around keys fucks with their order and ruins the zip() call later.

set() is unnecessary anyway.

Also remove enable/disable checking for USE (all flags in USE were enabled)

We should take IUSE.intersect(USE) in teh future but not yet implemented.

6 years agos/sydout/stdout/
Alec Warner [Sun, 18 May 2008 04:38:59 +0000 (21:38 -0700)]
s/sydout/stdout/

6 years agoUSE should be a list, not a string.
Alec Warner [Sun, 18 May 2008 04:37:35 +0000 (21:37 -0700)]
USE should be a list, not a string.

6 years agoAdd missing import
Alec Warner [Sun, 18 May 2008 04:35:27 +0000 (21:35 -0700)]
Add missing import

6 years agobackwards compat support for non-svn versions of portage.
Alec Warner [Sun, 18 May 2008 04:34:19 +0000 (21:34 -0700)]
backwards compat support for non-svn versions of portage.

6 years agouse catpkgsplit in the portage provider to get version information
Alec Warner [Sun, 18 May 2008 04:33:01 +0000 (21:33 -0700)]
use catpkgsplit in the portage provider to get version information

6 years agokeys should be lowercase I think ;)
Alec Warner [Sun, 18 May 2008 04:19:05 +0000 (21:19 -0700)]
keys should be lowercase I think ;)

6 years agoDoh, exclude this file.
Alec Warner [Sun, 18 May 2008 04:02:23 +0000 (21:02 -0700)]
Doh, exclude this file.

6 years agoAdd a simple client that can generate XML for us.
Alec Warner [Sun, 18 May 2008 04:00:57 +0000 (21:00 -0700)]
Add a simple client that can generate XML for us.

6 years agoLeave a note for handing POST data for CP so users can upload data.
Alec Warner [Mon, 12 May 2008 11:37:54 +0000 (04:37 -0700)]
Leave a note for handing POST data for CP so users can upload data.

Note for the future that per CP uploads are probably going to be slow and
any client will need to make hundreds of requests on average to add all
their packages.  Doing this over http is kind of crappy.

How about..

/api/host/

Lets you PUT and POST entire sets of data per host.

Would require storing host data though.

6 years agoClose <name> tag
Alec Warner [Mon, 12 May 2008 11:07:22 +0000 (04:07 -0700)]
Close <name> tag

6 years agoActually pass cp to CPOverview
Alec Warner [Mon, 12 May 2008 11:02:32 +0000 (04:02 -0700)]
Actually pass cp to CPOverview

6 years agoFix Typo
Alec Warner [Mon, 12 May 2008 11:01:41 +0000 (04:01 -0700)]
Fix Typo

6 years agoAdd XML results for CPOverview
Alec Warner [Mon, 12 May 2008 10:56:31 +0000 (03:56 -0700)]
Add XML results for CPOverview

6 years agoAdd CP method.
Alec Warner [Mon, 12 May 2008 10:51:07 +0000 (03:51 -0700)]
Add CP method.

6 years agoRemove html submit code.
Alec Warner [Mon, 12 May 2008 10:47:54 +0000 (03:47 -0700)]
Remove html submit code.

Rewrite /arch stuff to use 1 function and call out to individual functions
based on REQUEST_METHOD (GET,PUT, etc...)

6 years agoimport os instead of sys for kill and getpid
Alec Warner [Mon, 12 May 2008 03:45:03 +0000 (20:45 -0700)]
import os instead of sys for kill and getpid

6 years agoColubrid or paste catches sys.exit(), use sigkill
Alec Warner [Mon, 12 May 2008 03:43:29 +0000 (20:43 -0700)]
Colubrid or paste catches sys.exit(), use sigkill

6 years agoFIx routing.
Alec Warner [Mon, 12 May 2008 03:38:28 +0000 (20:38 -0700)]
FIx routing.

6 years agoFix death
Alec Warner [Mon, 12 May 2008 03:31:00 +0000 (20:31 -0700)]
Fix death

6 years agoAnother typo..
Alec Warner [Mon, 12 May 2008 03:27:28 +0000 (20:27 -0700)]
Another typo..

6 years agoServer doesn't die easily due to threading. Add a simple way to kill it.
Alec Warner [Mon, 12 May 2008 03:25:26 +0000 (20:25 -0700)]
Server doesn't die easily due to threading.  Add a simple way to kill it.

6 years agoAdd xml header. Set to text/plain for now.
Alec Warner [Mon, 12 May 2008 03:21:17 +0000 (20:21 -0700)]
Add xml header.  Set to text/plain for now.

6 years agoMake sure we return valid XML
Alec Warner [Mon, 12 May 2008 03:09:24 +0000 (20:09 -0700)]
Make sure we return valid XML

6 years agoFix loop in template to avoid NameError
Alec Warner [Mon, 12 May 2008 02:44:17 +0000 (19:44 -0700)]
Fix loop in template to avoid NameError

6 years agoRename var in arch_overview template so it is called properly
Alec Warner [Mon, 12 May 2008 02:42:22 +0000 (19:42 -0700)]
Rename var in arch_overview template so it is called properly

6 years agoAdd parenthesis to actually call the func.
Alec Warner [Mon, 12 May 2008 02:39:56 +0000 (19:39 -0700)]
Add parenthesis to actually call the func.

6 years agoSync up function names
Alec Warner [Mon, 12 May 2008 02:38:29 +0000 (19:38 -0700)]
Sync up function names

6 years agoOops, add keyword for status
Alec Warner [Mon, 12 May 2008 02:37:37 +0000 (19:37 -0700)]
Oops, add keyword for status

6 years agoAdd craptastic 404 page for now. What do REST methods do when they are invalid?
Alec Warner [Mon, 12 May 2008 02:36:16 +0000 (19:36 -0700)]
Add craptastic 404 page for now.  What do REST methods do when they are invalid?

6 years agoAdd per arch stats to reporter.
Alec Warner [Mon, 12 May 2008 02:31:15 +0000 (19:31 -0700)]
Add per arch stats to reporter.

Add per arch stats to arch/FOO request, top10 is unused right now (make the client do it!)

6 years agotype -> Type
Alec Warner [Mon, 12 May 2008 02:21:44 +0000 (19:21 -0700)]
type -> Type

6 years agoTry changing context type to text/xml
Alec Warner [Mon, 12 May 2008 02:18:11 +0000 (19:18 -0700)]
Try changing context type to text/xml

6 years agoRemove default args.
Alec Warner [Mon, 12 May 2008 02:02:52 +0000 (19:02 -0700)]
Remove default args.

cpvs -> cps

6 years agoAdd code for GetCP, comment out GetCPV route as we havent' added CPVs yet
Alec Warner [Mon, 12 May 2008 00:33:07 +0000 (17:33 -0700)]
Add code for GetCP, comment out GetCPV route as we havent' added CPVs yet

6 years agoRemove Index code.
Alec Warner [Mon, 12 May 2008 00:27:05 +0000 (17:27 -0700)]
Remove Index code.

6 years agoRemove code that used static/
Alec Warner [Mon, 12 May 2008 00:24:53 +0000 (17:24 -0700)]
Remove code that used static/

6 years agoPunt more HTML stuff and add CP and CPV overviews (no templates yet).
Alec Warner [Mon, 12 May 2008 00:23:25 +0000 (17:23 -0700)]
Punt more HTML stuff and add CP and CPV overviews (no templates yet).

We will use REQUEST_METHOD to serve GET requests for now, will proabbly end up serving some ghetto XML format.

6 years agoMove to a RESTFul interface with no actual HTML.
Alec Warner [Sun, 11 May 2008 21:55:23 +0000 (14:55 -0700)]
Move to a RESTFul interface with no actual HTML.

6 years agoAdd function to get HTTP request method.
Alec Warner [Sun, 11 May 2008 21:32:12 +0000 (14:32 -0700)]
Add function to get HTTP request method.

6 years agoTry to document what the * means.
Alec Warner [Sun, 11 May 2008 11:24:52 +0000 (04:24 -0700)]
Try to document what the * means.

6 years agoFix '' once and for all ;)
Alec Warner [Sun, 11 May 2008 11:10:44 +0000 (04:10 -0700)]
Fix '' once and for all ;)

6 years agoFix typo
Alec Warner [Sun, 11 May 2008 11:05:12 +0000 (04:05 -0700)]
Fix typo

6 years agoSomehow '' is getting in the testdata, remove it.
Alec Warner [Sun, 11 May 2008 10:57:54 +0000 (03:57 -0700)]
Somehow '' is getting in the testdata, remove it.

6 years agoAha, the reporting logic was using [0] to cut off extra cps
Alec Warner [Sun, 11 May 2008 10:55:55 +0000 (03:55 -0700)]
Aha, the reporting logic was using [0] to cut off extra cps
and since everything was tied with '1' installed for the test data
everything except for '' basically got cut off ;)

6 years agoPrevious change was wrong (bug is somewhere else).
Alec Warner [Sun, 11 May 2008 09:54:24 +0000 (02:54 -0700)]
Previous change was wrong (bug is somewhere else).

Also add documentation and do less cache checkpointing (track if we actually modified the cache).

6 years agoWas wondering why I had no data, now I know why, damn off by one errors ;)
Alec Warner [Sun, 11 May 2008 09:39:40 +0000 (02:39 -0700)]
Was wondering why I had no data, now I know why, damn off by one errors ;)

6 years agos/max/sum/
Alec Warner [Sun, 11 May 2008 09:27:10 +0000 (02:27 -0700)]
s/max/sum/

Moved files from uploaded/ to processed/ after we process them.

6 years agoPrint debug info regarding options.
Alec Warner [Sun, 11 May 2008 08:57:47 +0000 (01:57 -0700)]
Print debug info regarding options.

6 years agoAdd debugging logging for cache insertion.
Alec Warner [Sun, 11 May 2008 08:47:21 +0000 (01:47 -0700)]
Add debugging logging for cache insertion.

6 years agoAdd right_side template.
Alec Warner [Sun, 11 May 2008 08:45:31 +0000 (01:45 -0700)]
Add right_side template.

6 years agoAdd help and about templates.
Alec Warner [Wed, 19 Mar 2008 12:56:44 +0000 (05:56 -0700)]
Add help and about templates.

Add routing for help and about.

Add right_side set of links to bugs, forums, help, and about.

Add FAQ.

Add --mode, to help users identify that this is not a production application.

Flesh out footer.

Screw with css a bunch.

6 years agoGraaf said to use position:absolute, and I figured that made sense.
Alec Warner [Wed, 19 Mar 2008 10:27:04 +0000 (03:27 -0700)]
Graaf said to use position:absolute, and I figured that made sense.

6 years agoMake the logo link to '/'
Alec Warner [Wed, 19 Mar 2008 10:24:30 +0000 (03:24 -0700)]
Make the logo link to '/'

6 years agoOops, add missing template to git ;)
Alec Warner [Wed, 19 Mar 2008 10:21:45 +0000 (03:21 -0700)]
Oops, add missing template to git ;)

6 years agoAdd stylesheets.
Alec Warner [Tue, 18 Mar 2008 13:55:15 +0000 (06:55 -0700)]
Add stylesheets.

Center header image.

Add leftside <div>

Float left-side menu.

Make left side menu a bit more descriptive.

6 years agohostname=* doesn't work :)
Alec Warner [Tue, 18 Mar 2008 13:32:30 +0000 (06:32 -0700)]
hostname=* doesn't work :)

6 years agoNote that the cache moved.
Alec Warner [Tue, 18 Mar 2008 13:21:05 +0000 (06:21 -0700)]
Note that the cache moved.

6 years agoAdd support for --port and --hostname.
Alec Warner [Tue, 18 Mar 2008 13:10:22 +0000 (06:10 -0700)]
Add support for --port and --hostname.
Default hostname is 'localhost'.

Add hostname=* to flagfile

6 years agoSigh, fix missing % in logging statement
Alec Warner [Tue, 18 Mar 2008 13:06:42 +0000 (06:06 -0700)]
Sigh, fix missing % in logging statement

6 years agocatch IOError
Alec Warner [Tue, 18 Mar 2008 13:05:12 +0000 (06:05 -0700)]
catch IOError

6 years agoMore 2.5 -> 2.4 crap for python
Alec Warner [Tue, 18 Mar 2008 13:03:23 +0000 (06:03 -0700)]
More 2.5 -> 2.4 crap for python

6 years agoFix with_statement crap again.
Alec Warner [Tue, 18 Mar 2008 13:02:54 +0000 (06:02 -0700)]
Fix with_statement crap again.
Fix try finally block in processor.
Fix python version

6 years agoin templates: valid_arches -> arches
Alec Warner [Mon, 17 Mar 2008 11:10:45 +0000 (04:10 -0700)]
in templates: valid_arches -> arches

Add left side template.

Make reporting TopN detect keyerrors and return [].

Routes and Colubrid pass arch with / in front of it, so
/arches/x86 gives me /x86 as an arg, so strip the last / to get
the requested arch.

Make things use _GenDefaultArgs for header/footer/left_side info.

6 years agoAdd arch overview to GStatsApp,
Alec Warner [Mon, 17 Mar 2008 10:15:00 +0000 (03:15 -0700)]
Add arch overview to GStatsApp,
Add /arch/$ARCH mapping to gstats_frontend

Add testdata/

Add static/

Add mapping in frontend to serve items in /static.

Add wrong png to header.html.

Switch locking in GSDataCache to not use with_statement, as it requires python2.5

Fix bug where app would start and injector would try to read from the on-disk
datastore but the datastore did not exist yet.

Add reporting func to generate topN CPs for arches.

6 years agoMove GSDataCache to gstats_cache.py
Alec Warner [Thu, 13 Mar 2008 06:53:07 +0000 (23:53 -0700)]
Move GSDataCache to gstats_cache.py

Make GStatsProcessor a subclass of Thread instead of an App Timer.

Remove some overzeallous logging statements.

Make the cache save when it is updated.

Lock the cache with a loggingLock that wraps threading.Lock()

Require 2.5, because we are using with_statement from __future.

6 years agoAdd flagfile
Alec Warner [Tue, 11 Mar 2008 13:42:01 +0000 (06:42 -0700)]
Add flagfile

Modify arch_usage template slightly.

Add a thread to update the cache every X minutes.

add FLAGS.cache_path.

Add Data injector, logging statements.

Making reporting use in memory cache instead of on disk cache.

6 years agoAdd flagfile, --valid_archives, fix processor up a bit.
Alec Warner [Mon, 10 Mar 2008 15:10:55 +0000 (08:10 -0700)]
Add flagfile, --valid_archives, fix processor up a bit.

6 years agoHrm, I suck at git.
Alec Warner [Sun, 9 Mar 2008 20:50:57 +0000 (13:50 -0700)]
Hrm, I suck at git.

Also add README and uncommited changes to gstats_storage (see last commit msg for what changed)

6 years agoAdd gstats_processor a module that works on injecting uploaded/*
Alec Warner [Sun, 9 Mar 2008 20:45:19 +0000 (13:45 -0700)]
Add gstats_processor a module that works on injecting uploaded/*
files into the data cache.

The in memory data cache is also in that module for now.

gstats_storage.py, remove YieldAllArches as it was dumb and poorly implemented.

6 years agoInitial Import of gstats service.
Alec Warner [Sun, 9 Mar 2008 15:12:37 +0000 (08:12 -0700)]
Initial Import of gstats service.

the frontend.py takes care of routing (via dev-python/Routes).
the app.py is the common 'Controller' and takes care of gathering
the data and loading the templates or views.

The templates/ dir and Mako take care of display.

the storage.py handles storage for now; it is crap, needs locking; still trying
to avoid sql...