Administering a SVN repository

Contents

  1. Overview
  2. Granting read and write access to SVN - the authz file
  3. Support for E-groups in the authz file
  4. Versioning of configuration files
  5. Checking the syntax of the SVN authz file
  6. Configuring Trac
  7. Trac plugins
  8. Modifying configuration for commit emails
  9. Managing the hooks
  10. Make your repository work faster
  11. Granting (and removing) anonymous access
  12. Accessing a repository on a file-system level
  13. MIME properties of files
  14. Changing the repository on file-system level
  15. Permanently remove content from a repository
  16. Changing the password of the librarian service account
  17. Changing the person responsible for a librarian service account
  18. Changing the librarian service account
  19. AFS Disk quota
  20. Usage statistics
  21. Mirror your repository using svnsync

  1. Overview

    Once your project has been created, you can use the librarian service account to modify or configure the following:

    • E-groups:
      • E-groups can be used in the conf/authz file to control access to SVN. (Please see below.)
      • To modify egroups, go to the E-groups site.
    • Configuration files:
      • /afs/cern.ch/project/svn/reps/<project>/conf/authz: configures permissions for read and write access,
      • 'mailer.conf' and 'svn-mailer.conf' configure email notifications.
      • Hooks in /afs/cern.ch/project/svn/reps/<your_project>/usr-hooks/: triggers various actions at commit time.
      • Web interface Trac can be configured from the admin bar when logged in as librarian.
    • When logged-in as librarian on lxplus (ssh <librarian>@lxplus), you can run the svnadmin command on your repository (run svnadmin help for more details). This should only be done by experienced users.
  2. Managing Read and Write access to SVN - the authz file

    The person responsible for a given project is the librarian, and he/she does the administration work (e.g. controlling access). A librarian service account (e.g. libatlas for an atlas project) may be created for a project (see a list of repositories and librarians). The librarian account is allowed to modify everything in his/her project, including the configuration files in the repository, but modifying files directly in the repository should only be done by experienced users.

    Note that some experiments e.g. ATLAS, manage many of their atlas-<account> repositories using one librarian service account and have their own system for delegating access. If you are part of a group or experiment like this, please check with them first to understand how access is controlled.

    If you (the librarian) want to give write access to a given user, you will need to log on to lxplus using your librarian service account and modify the following file:

    /afs/cern.ch/project/svn/reps/<your_project>/conf/authz

    Put the account name of the user into the authz file following the syntax in the file. You may define groups and specify access per path. To enable (or restrict) read access for anyone with an account, add (or remove) the r:

    [/]
    * = r 
    

    which means that anyone with a CERN AFS account has read access. Similarly, to enable (or restrict) write access access for CERN account my_cern_account, add (or remove) the w:

    [/] 
    my_cern_account = rw

    Furthermore you can give access by path:

    [/] 
    * = r
    [/secret/path] 
    * = 
    svnadmin = rw 
    svnlemon = r 
    svnstats = r 
    jbond = rw 
    

    In the example above only the CERN account jbond has read and write access to the path: /secret/path in your repository. Note that you can also use groups, for example in the example above, replace the CERN account by the group you have defined:

    [groups] 
    secret = jbond, spiderman 
    [/] 
    * = r 
    [/secret/path] 
    * = 
    @secret = rw 
    svnadmin = rw 
    svnlemon = r 
    svnstats = r 
    

    Please always give access to the service accounts used by the CERN SVN Service in order to monitor and generate statistics etc. For more examples on how to manage the access file, see the commented lines in the file. This file controls access for svn clients, WebSVN and the Trac web interface. For more details on the authz file, see the SVN book here.

    If your repository is configured for open access (World), this means that anyone with an Internet connection can read all the data and history in your repository. If you wish to add or remove this feature, send an email to svn.support@cern.ch. to find out with what access your project is configured with, have a look here.

  3. Using E-groups in the SVN authz file

    Using the librarian account of your project, you can use E-groups to manage the access control of your Subversion repository. This is how it works:

    • In the group definition section of your authz file, you specify the names of the e-groups you want to use;
    • In the rest of your authz file, you use the e-groups like any other svn group defined in your authz file;
    • Several times a day each e-group member list is automatically expanded in the authz file.

    In order to avoid mistakes, we have automatically added the following section on top of the [groups] section of the authz file:

    # BEGIN EGROUPS DEFINITIONS 
    # my-group-name = 
    # END EGROUPS DEFINITIONS

    To use actual e-groups you have to add one e-group per line within the #BEGIN and #END block, with an empty member list. This empty list is automatically filled in by the e-group members. In the following example, we are using the e-groups Italians-at-cern and cern-staff, and the normal group developers to define access control:

    [groups] 
    # BEGIN EGROUPS DEFINITIONS
    Italians-at-cern = 
    cern-staff = 
    # END EGROUPS DEFINITIONS 
    developers = johnny, joey, deedee
    
    [/]
    @cern-staff = rw 
    @Italians-at-cern = r
    
    [/my/path] 
    @developers = rw
    
    Warnings:
    • The E-groups are expanded only if their definitions are between the special comment lines
      # BEGIN EGROUPS DEFINITIONS

      and

      # END EGROUPS DEFINITIONS

      , and if these lines are in the [groups] section of your authz file.

    • The E-groups are automatically filled 3 times a day, at 12h00, 16h00 and 19h00. See admin repository to have it filled when the authz file is changed.

    The recommended method for adding users to an e-group is through the egroups web interface. Only the main user account will be extracted (in case of a user having several accounts). If a specific account is needed, there is the possibility to add it explicitly, just change the 'Type' from 'Person' to 'Account' in the e-groups interface.

    If you put your authz file in the admin repository it will automatically be updated when you commit.

  4. Versioning of configuration files and hook scripts - the admin repository

    In Subversion, by default, your configuration files and hooks are not in your actual repository so changes to them are not versioned. If you want to use (SVN) version control on anything from the conf directory or usr-hooks directory, you may use a directory in the admin repository. The mechanism behind this is the following:

    • you store your configuration files and hooks in a directory of the special "administration" repository, called admin;
    • when you commit a new version to your directory in this admin repository, the files will be copied to your SVN repository with a post-commit hook.

    In order to start using this you have to:

    • checkout svn.cern.ch/reps/admin/<project> with the Subversion client and access method you prefer (HTTPS, SVN+SSH);
    • copy your current authz file from /afs/cern.ch/project/svn/reps/<project>/conf/authz to the conf/ directory and add it to the repository;
    • copy your hooks in the usr-hooks/ directory and add them to the repository;
    • commit.

    Warning: the admin post-commit hook will overwrite the actual files in your repository, so it is dangerous to modify both the admin repository and your repository files at the same time.

    The librarian service account has access to change your admin directory (svn.cern.ch/reps/admin/your_project_name). Older SVN repositories were setup with an egroup called VC-librarians-<project> and the members of this egroup also had access to change the admin folder. If your SVN repository does not have this egroup defined, then your librarian should define this egroup and send an email to svn.support@cern.ch requesting that this egroup is added to the SVN admin/conf/authz file in order to give access to your admin/<project> folder.

  5. Checking the syntax of the SVN authz file

    If the syntax off the SVN authz file is wrong, the repository may not work or access might be granted in another way than expected. Every day the syntax of all authz files are automatically checked. An email is sent to the librarian accounts telling them whenever the syntax is wrong. The syntax check includes:

    • Duplicate group detection in group section
    • Incorrect group name description
    • Incorrect group content
    • Missing sections

    The librarian can check the syntax of the authz file manually by using the script:

    /afs/cern.ch/user/s/svnadmin/public/authz-valid <path-of-authz>
    

    Note that normally only the librarian service account has file access for this check.

  6. Configuring Trac (not setup for new SVN repositories)

    Note that Trac is no longer setup by default for new SVN repositories as the recommended ticket system is JIRA. However if you wish to use Trac then send a request to svn.support@cern.ch

    To manage permissions and configurations in Trac, log on to Trac using the svn librarian account, you then have a menu bar called Admin. Under General/Permissions you can specify who has access (specify account names) and to what. You can also define groups. Under Plugins you can enable and disable available plugins. In the trac.ini you can customize everything including added plugins. On the svn and svnweb servers The Trac project environment is located at /reps/<your_project>/trac

    You can do many things using Trac, for example you can send notification emails using Trac, for more details, have a look here: Trac Notification, basically what you need to change is the following in the Admin section conf/trac.ini, notification:

    admit_domains = cern.ch 
    always_notify_owner = true 
    always_notify_reporter = true 
    smtp_default_domain = cern.ch 
    smtp_enabled = true 
    smtp_from = trac@whateverthatdoesnotexist.com 
    smtp_replyto = donotreply@cern.ch
    

    In order to debug trac you can set the log_level in the logging section in to DEBUG and the log_type to file. Next set the log to point to (for example):

    /afs/cern.ch/project/svn/reps/<your_project>/logs/trac.log

    Do not forget to switch back to WARNING when you are finished debugging.

    Trac uses Enscript for colour syntax. To customize, have a look on the mimeviewer section. For example to change .m files to be coloured as matlab files, add the following in the mimeviewer,mime_map section (add a comma before if you add it last in the line):

    text/x-matlab:m 
  7. Trac plugins

    For Trac there are plenty of Plugins available, however, for security reasons librarians are not allowed to upload plugins themselves, but may request plugins (or other features) here. Note Trac and its Plugins are provided on an as-is basis. The Version Control Team does not have the resources to give help or debug these, but may be able to help (at low priority) installing new versions if there are problems.

    The following Plugins are currently available (in order to enable them see below):

    Plugin name Official Link Default/optional Comment
    IniAdmin 0.2 Iniadmin default Used to manage Trac configurations
    TracFullBlogPlugin 0.1 FullBlog optional Create Blogs
    TracRevtreePlugin 0.6.3 RevTree optional Tree/graph of revisions. By default it works with repository layout with /trunk at the root, if you have a multiple-project repository layout, you need to change branch_re to
    ^(?P<branch>[^/]+/(?:(?:tags|branches)/[^/]+|trunk))(?:/(?P<path>.*))?$

    in the revtree admin section, see RevTree

    RevTree MergeInfoEnhancer MergeInfoEnhancer optional Plugin for RevTreePlugin that analyses the properties that are added by svn on merge operations
    XML-RPC XML-RPC optional Allows Trac plugins to export select parts of their interface via XML-RPC. Once the plugin is enabled you have to grant XML-RPC permissions to developers, via Trac's "Admin"->"Permissions".
    NB: If your client does not support the CERN login, you may need to grant XML-RPC permissions to "anonymous" as indicated in the XmlRpcPlugin documentation. (The former non_sso url is now depricated for security reasons.)

    Do not add any other plugins not documented here. In order to add or remove a Plugin, login to Trac as librarian, next click on Admin in the horizontal menu. Finally click on Plugins under General in the vertical left Administration menu bar. click on the plugin you want to add or remove and tick or untick the boxes. Click Apply changes. To customize the plugins, have a look at appropriate link above, you may also consult Trac.

  8. Modifying configuration for commit emails

    To configure who receives email commit notification, via the svn hooks (this can also be accomplished using trac), change the following file(s).

    • /afs/cern.ch/project/svn/reps/<your_project>/conf/mailer.conf
    • /afs/cern.ch/project/svn/reps/<your_project>/conf/svn-mailer.conf
    • /afs/cern.ch/project/svn/reps/<your_project>/usr-hooks/post-commit

    The mailer.conf is for mailer.py and svn-mailer is for svn-mailer, by default they are both commented out. To enable one of the commit email script, simply un-comment the corresponding line in usr-hooks/post-commit for the script you wish to use, for details about there speed see see the tables below (mailer.py is faster). To add an email address, simply find the following section and add all emails separated with white space (not commas):

    to_addr = <your_project>.SVNLibrarian@cern.ch some@email.com
    			
  9. Managing the hooks

    The following hooks are used, they are named according to their function:

    • post-commit.tmpl
    • post-lock.tmpl
    • post-revprop-change.tmpl
    • post-unlock.tmpl
    • pre-commit.tmpl
    • pre-lock.tmpl
    • pre-revprop-change.tmpl
    • pre-unlock.tmpl
    • start-commit.tmpl

    Some hooks are enabled by default, to learn more in general about how to enable hooks and what hooks you may implement, see the following web pages:

    In CERN's Central SVN service, for security reasons when a hook script is executed, it is run in a jailed environment to prevent a malicious hook accessing any other svn repository or system files. From a user's point of view it looks the same with a few minor changes:

    • the hooks reside in /afs/cern.ch/project/svn/reps/<your_project>/usr-hooks
    • You may only run bash, sh shell scripts, perl or python, all other scripts and languages are not supported.
    • Some dummy scripts are already setup so that lines may be uncommented to activate actions.
  10. Make your repository work faster

    If you find your repository is slow, possible improvements are:

    • The post-commit email notification is quite long, in order to disable it, comment out the line(s) with svn-mailer or/and mailer.py in
      /afs/cern.ch/project/svn/reps/<your_project>/usr-hooks/post-commit
    • We strongly recommend not putting binary files of any sort into SVN because they cost space as frequently a complete new version is stored for each change. So committing binary files, compress them first.

    If you disable the hooks there are other ways to get notification when a commit is made, for example rss feed or Trac.

  11. Granting (and removing) anonymous (guest) access

    If you wish to add or remove anonymous or guest access (accessible from the whole world) to your repository (that is accessible from the whole world), send an email to svn.support@cern.ch.

    Note that the guest access only enables read access (which is probably what you want anyway). If you have guest access enabled you may access it using

    http://svn.cern.ch/guest/<your_project>

    for example try:

    svn co http://svn.cern.ch/guest/<your_project>
  12. Accessing a repository on a file-system level

    Sometimes you need access to your repository on a file-system level, for example to run the svnadmin command. A repository hosted in the CERN Central SVN Service is available via AFS, at /afs/cern.ch/project/svn/reps/<project>. Only the librarian service account of each project has file access, so you will need to log in using the librarian service account on lxplus or wherever you have Subversion >= 1.5.

    Always use the svn tools when modifying/correcting your repository as the files are stored in a database and are not visible as normal files. The available commands and all their sub commands are:

    svn help 
    svnadmin help 
    svnsync help 
    svndumpfilter help 
    svnlook help

    Note that if you want to remove the whole project, you have to contact the SVN Service (send an e-mail to svn.support@cern.ch)

  13. MIME properties of files

    Setting properties of files/directories is a client-side action, since it requires a commit. In order to automate the action, you could use the enable-auto-props option in your client's config file.

    Some nice examples for how you can set the svn auto-props configurations can be found here.

    The most suggested solutions seem to be server-side enforcement, through a pre-commit hook that aborts the commit if a mime-type is not set.

    This could be the solution but there is a drawback: MIME types are not 100% standardized (see RFC 2045 MIME - part one). Therefore our suggestion is to start a discussion with your librarian and developers, in order to:

    • agree a set of standard MIME Types to be used in the repository;
    • create a standard [auto-props] section for client configuration (making sure you cover all the clients in use/supported);
    • use a pre-commit hook to check and reject commits that do not follow your standard.

     

  14. Changing the repository on a file-system level

    If you want to modify the content of your repository at a file system level (for example you may not be happy with the already loaded cvs conversion) you might try and delete your repository and create a new one, but at least two things would fail, firstly you do not have the permissions to delete the hooks directory. Secondly some of settings created by CERN's SVN Service in the conf directory and usr-hooks would be lost. Besides the trac environment may need to be re-synchronized. The following tool maintains the configure fails and re-synchronize Trac:

    $ cd /afs/cern.ch/project/svn/lib 
    $ ./inject_svn_dumpfile.sh
    This script replaces a repository with a new repository 
    keeping the config files and hook scripts in CERN Central SVN service.  
    
    Usage: 
    	inject_svn_dumpfile
    	project - short name of the project, small letters,
    				no spaces, ex: atlas
     	svn_dumpfile - path to new subversion repository
    
  15. Permanently remove content from a repository

    In subversion all files ever committed into the repository are to be found either in historical revisions or in the latest revision. Sometimes this is not desirable. Unfortunately there is no subversion command that deals with this issue and you have to delete the unwanted content yourself following this procedure:

    svnadmin dump /afs/cern.ch/project/svn/reps/<your_project> > dmpfile
    svndumpfilter exclude <path to file> < dmpfile > filtered-dmpfile 
    /afs/cern.ch/project/svn/lib/inject_svn_dumpfile.sh <your_project> filtered-dmpfile 
    
  16. Changing the password of the librarian service account

    The librarian service account is a service account belonging to the librarian. So to manage his accounts, the librarian uses the account management page where it also explains what to do if the password has been lost.

  17. Changing the person responsible for a librarian service account

    Sometimes the person responsible for a given SVN repository (that is the librarian) changes - for example when that person leaves or moves on to other work. The new librarian (person) will normally take over the same librarian service account (often called lib<your_project>). To change the person responsible for an SVN project the new person has to take over the librarian account. So the former librarian (person) should:

    1. set the forwarding of the librarian account's e-mail address to the new person (go to MMM web page - Email forwarding). For some SVN repositories an egroup has been used - and you may need to transfer the ownership at E-groups to the new librarian (and add him to the members of the egroup), the name of the e-group is:
      VC-librarians-<your_project>
    2. Give the password of the librarian MAIL/AFS service account to the new person.
    3. Transfer the ownership of the librarian service account to the new person, using the account management page.
  18. Changing the librarian service account

    The librarian account is a service account (often of the form lib<your_project>), belonging to the person who is the project librarian. Usually, this account stays with the project for the life of the project. A librarian service account may be passed to another person when the librarian changes (see above).

    However, a change of librarian account to a different account implies some modification in the SVN Service configuration files. To request it, the former librarian should send a message to SVN.Support@cern.ch, naming the new librarian account and the person who will become the new librarian.

  19. AFS Disk quota

    SVN repositories are hosted on AFS with each repository in a seperate AFS voume. The initial disk quota for a repository is 50MB. This quota will be automatically increased as the project grows. If you plan to quickly add a lot of files to a repository, please first send an e-mail to SVN.Support@cern.ch requesting how much additional disk quota you need.

  20. Usage statistics

    Usage statistics of a given SVN repository are available to everyone with a CERN account. Only projects with more than 10 revisions get statistics generated. Depending on the number of revisions and commit activity, different revision samples are used:

    • less than 10 revisions; no graphs
    • between 10 and 1099 revisions; sample: All
    • less than 20 commits a day and at least 1100 revisions; sample: All, last 500 and last 100 revisions
    • more than 20 commits a day and at least 6000 revisions; sample: All, last 5000 and last 1000 revisions

    Note that adding old migrated modules will make the graphs look strange on small samples since the time axis (x) gets very long. Furthermore "All" revisions is as many revisions that the statistic generation program finds suitable.

  21. Mirror your repository using svnsync

    In order to create a copy of your subversion project on another server, use the svnsync tool which comes with the subversion package. How svnsync and should be used is well documented in the subversion book. It is recommended to allow only one user, "svnsync", to commit into the mirror repository (if someone else commits the mirror will break). Using the hooks, only user svnsync is allowed to commit to the (empty) destination repository:

    svnadmin create mirror_rep 
    cd mirror_rep 
    cp hooks/pre-revprop-change.tmpl hooks/pre-revprop-change 
    cat << 'EOF' > hooks/pre-revprop-change 
    				
    #!/bin/sh 
    USER="$3"
    
    if [ "$USER" = "svnsync" ]; then exit 0; fi 
    
    echo "Only the svnsync user can change revprops" >&2 
    exit 1 
    EOF
    
    chmod +x hooks/pre-revprop-change
    
    cp hooks/start-commit.tmpl hooks/start-commit 
    cat << 'EOF' > hooks/start-commit 
    				
    #!/bin/sh
    
    USER="$2"  
    if [ "$USER" = "svnsync" ]; then exit 0; fi
    				
    echo "Only the svnsync user can change revprops" >&2 
    exit 1 
    EOF
    
    chmod +x hooks/start-commit
    

    Now we are ready to run svnsync. First run svnsync init followed by svnsync sync (svnsync sync could be run in a cron job from user svnsync in order to keep the mirror up to date).

    svnsync init file://`pwd` svn+ssh://svn.cern.ch/reps/<your_project> 
    Copied properties for revision 0 
    svnsync sync file://`pwd` 
    Committed revision 1. 
    Copied properties for revision 1. 
    Committed revision 2. 
    Copied properties for revision 2. 
    Committed revision 3. 
    Copied properties for revision 3. 
    ...